THE dark corners of cyberspace are being illuminated by indexing software that can reach into secretive websites that are normally inaccessible to search engines. This could allow search engines to cover online forums lurking within the 鈥渄ark web鈥�, and provide insights into what is being said by groups who would rather keep their conversations secret.
Conventional search engines use programs called spiders or web crawlers that scuttle around the internet and index what they find. However, many websites are protected by security restrictions that fend off such software. Screening out all traffic from IP addresses belonging to well-known search engines is one way to do this.
The dark web can provide a haven for extremist groups to exchange ideas, says , director of the artificial intelligence laboratory at the University of Arizona in Tucson. So Chen and his team devised software to access and index protected online forums (Journal of the American Society for Information Science and Technology, ).
Advertisement
One of the tricks deployed by Chen鈥檚 software is to regularly change the apparent IP address of the computer on which it is running. The software also disguises its indexing activity by making it look like the traffic generated by users browsing the forum. What鈥檚 more, it can attempt to sign up for membership on forums that require registration, though it has to seek help from Chen鈥檚 team if unusual information is asked for. To help it index text in languages other than English it uses , Google鈥檚 online translation engine.
鈥淭he software disguises its indexing activity to look like traffic generated by users browsing the forum鈥�
Unlike a regular web crawler, Chen鈥檚 software looks only at sites he has specified. It has compiled data on 29 restricted forums, containing about 13 million messages in total. On one forum, it took just 39 minutes to index 29,016 posts made over a six-week period.
Chen鈥檚 team is now analysing the conversations on these forums to build an overview of the links between participants. He suggests this may be useful in identifying prominent members.
The impressive thing about Chen鈥檚 forum crawler is the way it combines human guidance and automated web searches to catalogue dark web forums, says Denis Roy, a spokesman for Yahoo. 鈥淭he name of the game,鈥� he says, is to 鈥渇ind the right blend of the least possible number of humans and machines鈥� to perform this indexing of restricted websites efficiently.
