Robot access log

This page show you how to analyze the access log with the source code.

The function of this page
Location of Source code
Import access log

Start importing
Start back ground job with the progress bar
Framework of analyzing log

Log analyze policy

Source code of analyze
Detect the log is robot access or not

The function of this page

This page imports and analyze access log in order to detect robot access. By learning this, you can customize the way to detect robot and analytics.

Robot access log

About the detail, take a look at Checking the crawl and Import access log.

Location of Source code

Source code of this page is in the "[ALINOUS_HOME]/admin/robotlog/" folder.

location of the source code

Import access log

You can import access log of the http server from this page.

Import form

Choose file with the "Choose File" button and push "Import" button. Then progress bar apears and operation starts. About implementation of th progress bar, please take a look at jQuery Progress Bar.

Start importing

If you push the "import" button, the "$IN.cmd" parameter set as "uploadlog", and source code below executed.


include("/admin/robotlog/loganalyze.alns");
include("/include/progress.alns");


initParams($IN, $SESSION);

if($IN.cmd == "uploadlog"){
	uploadlog($IN, $SESSION);
}
else if($IN.cmd == "back"){
	redirect "/", 302;
}

// progress bar
$showProgress = isRunning($IN, $SESSION);

return 0;

In this code, it calls "uploadlog()", and the code is below.


function uploadlog($IN, $SESSION)
{
	if(ProgressJob.jobStarted("importrobot")){
		return 0;
	}
	
	$logfile = "/tmp/" + $SESSION + "_" + $IN.csvfile;
	$jobMax = countLines($logfile);
	
	$params.IN = $IN;
	$params.SESSION = $SESSION;
	ProgressJob.startJob("importrobot", $jobMax, "backbroundWrapper", $params);
	
	return 0;
}

Start back ground job with the progress bar

The "ProgressJob.startJob()" function starts back ground job with the progress bar. The source code is at"[ALINOUS_HOME]/include/progress.alns"

progress bar library

The source code is below.


function ProgressJob.startJob($jobId, $jobMax, $jobFunction, $params)
{
	synchronized{
		BEGIN;
		
		INSERT INTO job_progress(
			job_id,
			job_done,
			job_max
		)
		VALUES(
			$jobId,
			0, // job_done,
			$jobMax
		);
		
		COMMIT;		
	}
	
	$params.jobMax = $jobMax;
	Thread.execute($jobFunction, $params);
}

This code does operation below.

Insert record into the table which manage the progress bar status
Call Thread.execute() in order to start back ground job

Framework of analyzing log

After new thread to analyze access log launched by the main thread, next code is executed as back ground job.


function backbroundWrapper($params)
{
	$tmpFile = "/tmp/" + $params.SESSION + "_" + $params.IN.csvfile; 
	doUploadlog($tmpFile);
	
	ProgressJob.jobFinished("importrobot");
	
	return 0;
}

The main routine of log analyzing is in the doUploadlog() function. After executed the function, call "ProgressJob.jobFinished()" and finish the progress bar.

The source code of the main logic is below.


function doUploadlog($tmpFile)
{
	$bulknum = 512;
	$config = AccessLog.readConfig();
	
	AccessLogLineReader.openFile($tmpFile);
	
	$logline = AccessLogLineReader.readLine($tmpFile);
	while($logline != null){
		$idx = Array.size(@loglines);
		$loglines[$idx] = $logline;
		
		if($idx == $bulknum){
			bulkExecute(@loglines, $config);
			Variable.release(@loglines);
		}
		
		$logline = AccessLogLineReader.readLine($tmpFile);
	}
	
	bulkExecute(@loglines, $config);
	
	AccessLogLineReader.closeFile($tmpFile);
	
	File.remove($tmpFile);
}

function bulkExecute(@loglines, $config)
{
	$size = Array.size(@loglines);
	for($i = 0; $i < $size; $i++){
		$logline = $loglines[$i];
		$sync = parallel(18){
			local $logline;
			
			try{
				AccessLog.analyze($logline, $config);
			}catch($e){}
		};
	}
	
	Parallel.join($sync);
	
	// bulk update of the progressbar
	ProgressJob.progress($size, "importrobot");
}

The "AccessLog.readConfig()" function get access log format, and read some lines. After reading lines defined by "$bulknum" variable, analyze and insert the result simultaneously. In order to do this, the "bulkExecute()" function uses "Parallel execution" of Alinous-Core programming language.

Log analyze policy

The log analysing is done when you import access log. The function "AccessLog.analyze()" includes the policy of analyzing the access log.

When you customize the way to analyze the access log, please customize here.

Source code of analyze

The source code of "AccessLog.analyze()" is below.


function AccessLog.analyze($logline, $config)
{
	@logArray = AccessLogLineReader.parseLine($logline);
	
	$isRobot = AccessLog.isRobot(@logArray, $config);
	if(!$isRobot){
		return 0;
	}
	
	$remoteHost = $logArray[$config.remoteHost];
	$remoteHost = AccessLog.resolveIp($remoteHost);
	$access_time =  $logArray[$config.time];
	$access_time = AccessLogLineReader.parseTime($access_time);
	
	$robot_type = "googlebot";
	
	$user_name = "";
	$path = "";
	$protocol = "";
	$method = "";
	
	$requestFirstLine = $logArray[$config.requestFirstLine];
	@firstlines = String.split($requestFirstLine, " ");
	
	if($firstlines[1] != null){
		$path = $firstlines[1];
	}
	if($firstlines[0] != null){
		$method = $firstlines[0];
	}
	if($firstlines[2] != null){
		$protocol = $firstlines[2];
	}
	
	$bytes = 0;
	$referer = "";
	if($config.referer >= 0){
		$referer = $logArray[$config.referer];
	}
	$response_status = "";
	if($config.status >= 0){
		$response_status = $logArray[$config.status];
	}
	
	$user_agent = "";
	if($config.userAgent >= 0){
		$user_agent = $logArray[$config.userAgent];
	}
	
	BEGIN;
	
	SELECT * INTO wmc_robot_access FROM wmc_robot_access
	WHERE
		wmc_robot_access.remote_host = $remoteHost AND
		wmc_robot_access.access_time = $access_time;
	
	if(Array.size(@wmc_robot_access) == 0){
		INSERT INTO wmc_robot_access(
			remote_host,
			access_time,
			robot_type,
			user_name,
			path,
			protocol,
			method,
			bytes,
			referer,
			response_status,
			user_agent
		)
		VALUES(
			$remoteHost,
			$access_time,
			$robot_type,
			$user_name,
			$path,
			$protocol,
			$method,
			$bytes,
			$referer,
			$response_status,
			$user_agent
		);
	}
	else{
		UPDATE wmc_robot_access
		SET
			robot_type = $robot_type,
			user_name = $user_name,
			path = $path,
			protocol = $protocol,
			method = $method,
			bytes = $bytes,
			referer = $referer,
			response_status = $response_status,
			user_agent = $user_agent
		WHERE
			wmc_robot_access.remote_host = $remoteHost AND
			wmc_robot_access.access_time = $access_time;
	}
	
	COMMIT;
	

}

This function does operations below.

Check the access log is robot access or not
If the access is of robot, insert access log record

Detect the log is robot access or not

This function actually checks the log is robot access or not.


function AccessLog.isRobot(@logArray, $config)
{
	// check useragent
	if($config.userAgent >= 0){
		$userAgent = $logArray[$config.userAgent];
		
		if(String.indexOf($userAgent, "Googlebot", 0) < 0){
			return false;
		}
		if(String.indexOf($userAgent, "http://www.google.com/bot.html", 0) < 0){
			return false;
		}	
	}
	
	// checkIP
	$remoteHost = $logArray[$config.remoteHost];
	$hostName = AccessLog.resolveIp($remoteHost);// crawl-66-249-74-226.googlebot.com rate-limited-proxy-209-85-238-226.google.com
	
	$crawl = String.match($hostName, "crawl\-.*.googlebot.com");
	$rateLimited = String.match($hostName, "rate\-limited\-proxy\-.*.google.com");
	
	
	if($crawl == false && $rateLimited == false){
		$ipmatch = String.startsWith($hostName, "66.249.70.");
		if(!$ipmatch){
			return false;
		}
	}
	
	return true;
}

The policy to check it is below.

If access log format contains user agent, check the useragent, at first
Check the remote host IP's name