taixingyiji/openit
1
0
Fork 0
mirror of https://github.com/taixingyiji/openit.git synced 2025-11-01 21:22:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Delete utils/rm directory

Browse Source
This commit is contained in:
Yu Steven
2022-07-19 19:02:36 +08:00
committed by GitHub
parent 33cf048ef0
commit 79aec528f5
1653 changed files with 0 additions and 146883 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
utils/rm/node_modules/geoip-lite/LICENSE generated vendored
View File

@@ -1,50 +0,0 @@
There are two licenses, one for the software library, and one for the data.
This product includes GeoLite data created by MaxMind, available from http://maxmind.com/
SOFTWARE LICENSE (Node JS library)
The node-geoip JavaScript library is licensed under the Apache License, Version 2.0:
Copyright 2011 Philip Tellis <philip@bluesmoon.info>
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
DATABASES LICENSE (GeoLite2 databases)
Copyright (c) 2012-2018 MaxMind, Inc. All Rights Reserved.
The GeoLite2 databases are distributed under the
Creative Commons Attribution-ShareAlike 4.0 International License (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
https://creativecommons.org/licenses/by-sa/4.0/legalcode
The attribution requirement may be met by including the following in all
advertising and documentation mentioning features of or use of this
database:
This product includes GeoLite2 data created by MaxMind, available from
<a href="http://www.maxmind.com">http://www.maxmind.com</a>.
THIS DATABASE IS PROVIDED BY MAXMIND, INC ``AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL MAXMIND BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
DATABASE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

229
utils/rm/node_modules/geoip-lite/README.md generated vendored
View File

@@ -1,229 +0,0 @@
GeoIP-lite
==========
A native NodeJS API for the GeoLite data from MaxMind.
This product includes GeoLite data created by MaxMind, available from http://maxmind.com/
[![Build Status](https://travis-ci.org/bluesmoon/node-geoip.svg?branch=master "node-geoip on Travis")](https://travis-ci.org/bluesmoon/node-geoip)
introduction
------------
MaxMind provides a set of data files for IP to Geo mapping along with opensource libraries to parse and lookup these data files.
One would typically write a wrapper around their C API to get access to this data in other languages (like JavaScript).
GeoIP-lite instead attempts to be a fully native JavaScript library. A converter script converts the CSV files from MaxMind into
an internal binary format (note that this is different from the binary data format provided by MaxMind). The geoip module uses this
binary file to lookup IP addresses and return the country, region and city that it maps to.
Both IPv4 and IPv6 addresses are supported, however since the GeoLite IPv6 database does not currently contain any city or region
information, city, region and postal code lookups are only supported for IPv4.
philosophy
----------
I was really aiming for a fast JavaScript native implementation for geomapping of IPs. My prime motivator was the fact that it was
really hard to get libgeoip built for Mac OSX without using the library from MacPorts.
why geoip-lite
--------------
`geoip-lite` is a fully JavaScript implementation of the MaxMind geoip API. It is not as fully featured as bindings that use `libgeoip`.
By reducing scope, this package is about 40% faster at doing lookups. On average, an IP to Location lookup should take 20 microseconds on
a Macbook Pro. IPv4 addresses take about 6 microseconds, while IPv6 addresses take about 30 microseconds.
synopsis
--------
```javascript
var geoip = require('geoip-lite');
var ip = "207.97.227.239";
var geo = geoip.lookup(ip);
console.log(geo);
{ range: [ 3479298048, 3479300095 ],
country: 'US',
region: 'TX',
eu: '0',
timezone: 'America/Chicago',
city: 'San Antonio',
ll: [ 29.4969, -98.4032 ],
metro: 641,
area: 1000 }
```
installation
------------
### 1. get the library
$ npm install geoip-lite
### 2. update the datafiles (optional)
Run `cd node_modules/geoip-lite && npm run-script updatedb license_key=YOUR_LICENSE_KEY` to update the data files. (Replace `YOUR_LICENSE_KEY` with your license key obtained from [maxmind.com](https://support.maxmind.com/hc/en-us/articles/4407111582235-Generate-a-License-Key))
You can create maxmind account [here](https://www.maxmind.com/en/geolite2/signup)
**NOTE** that this requires a lot of RAM. It is known to fail on on a Digital Ocean or AWS micro instance.
There are no plans to change this. `geoip-lite` stores all data in RAM in order to be fast.
API
---
geoip-lite is completely synchronous. There are no callbacks involved. All blocking file IO is done at startup time, so all runtime
calls are executed in-memory and are fast. Startup may take up to 200ms while it reads into memory and indexes data files.
### Looking up an IP address ###
If you have an IP address in dotted quad notation, IPv6 colon notation, or a 32 bit unsigned integer (treated
as an IPv4 address), pass it to the `lookup` method. Note that you should remove any `[` and `]` around an
IPv6 address before passing it to this method.
```javascript
var geo = geoip.lookup(ip);
```
If the IP address was found, the `lookup` method returns an object with the following structure:
```javascript
{
range: [ <low bound of IP block>, <high bound of IP block> ],
country: 'XX', // 2 letter ISO-3166-1 country code
region: 'RR', // Up to 3 alphanumeric variable length characters as ISO 3166-2 code
// For US states this is the 2 letter state
// For the United Kingdom this could be ENG as a country like “England
// FIPS 10-4 subcountry code
eu: '0', // 1 if the country is a member state of the European Union, 0 otherwise.
timezone: 'Country/Zone', // Timezone from IANA Time Zone Database
city: "City Name", // This is the full city name
ll: [<latitude>, <longitude>], // The latitude and longitude of the city
metro: <metro code>, // Metro code
area: <accuracy_radius> // The approximate accuracy radius (km), around the latitude and longitude
}
```
The actual values for the `range` array depend on whether the IP is IPv4 or IPv6 and should be
considered internal to `geoip-lite`. To get a human readable format, pass them to `geoip.pretty()`
If the IP address was not found, the `lookup` returns `null`
### Pretty printing an IP address ###
If you have a 32 bit unsigned integer, or a number returned as part of the `range` array from the `lookup` method,
the `pretty` method can be used to turn it into a human readable string.
```javascript
console.log("The IP is %s", geoip.pretty(ip));
```
This method returns a string if the input was in a format that `geoip-lite` can recognise, else it returns the
input itself.
Built-in Updater
----------------
This package contains an update script that can pull the files from MaxMind and handle the conversion from CSV.
A npm script alias has been setup to make this process easy. Please keep in mind this requires internet and MaxMind
rate limits that amount of downloads on their servers.
You will need, at minimum, a free license key obtained from [maxmind.com](https://support.maxmind.com/hc/en-us/articles/4407111582235-Generate-a-License-Key) to run the update script.
Package stores checksums of MaxMind data and by default only downloads them if checksums have changed.
### Ways to update data ###
```shell
#update data if new data is available
npm run-script updatedb license_key=YOUR_LICENSE_KEY
#force udpate data even if checkums have not changed
npm run-script updatedb-force license_key=YOUR_LICENSE_KEY
```
You can also run it by doing:
```bash
node ./node_modules/geoip-lite/scripts/updatedb.js license_key=YOUR_LICENSE_KEY
```
### Ways to reload data in your app when update finished ###
If you have a server running `geoip-lite`, and you want to reload its geo data, after you finished update, without a restart.
#### Programmatically ####
You can do it programmatically, calling after scheduled data updates
```javascript
//Synchronously
geoip.reloadDataSync();
//Asynchronously
geoip.reloadData(function(){
console.log("Done");
});
```
#### Automatic Start and stop watching for data updates ####
You can enable the data watcher to automatically refresh in-memory geo data when a file changes in the data directory.
```javascript
geoip.startWatchingDataUpdate();
```
This tool can be used with `npm run-script updatedb` to periodically update geo data on a running server.
Caveats
-------
This package includes the GeoLite database from MaxMind. This database is not the most accurate database available,
however it is the best available for free. You can use the commercial GeoIP database from MaxMind with better
accuracy by buying a license from MaxMind, and then using the conversion utility to convert it to a format that
geoip-lite understands. You will need to use the `.csv` files from MaxMind for conversion.
Also note that on occassion, the library may take up to 5 seconds to load into memory. This is largely dependent on
how busy your disk is at that time. It can take as little as 200ms on a lightly loaded disk. This is a one time
cost though, and you make it up at run time with very fast lookups.
### Memory usage ###
Quick test on memory consumption shows that library uses around 100Mb per process
```javascript
var geoip = require('geoip-lite');
console.log(process.memoryUsage());
/**
* Outputs:
* {
* rss: 126365696,
* heapTotal: 10305536,
* heapUsed: 5168944,
* external: 104347120
* }
**/
```
Alternatives
----------
If your use-case requires doing less than 100 queries through the lifetime of your application or if you need really fast latency on start-up, you might want to look into [fast-geoip](https://github.com/onramper/fast-geoip) a package with a compatible API that is optimized for serverless environments and provides faster boot times and lower memory consumption at the expense of longer lookup times.
References
----------
- <a href="http://www.maxmind.com/app/iso3166">Documentation from MaxMind</a>
- <a href="http://en.wikipedia.org/wiki/ISO_3166">ISO 3166 (1 & 2) codes</a>
- <a href="http://en.wikipedia.org/wiki/List_of_FIPS_region_codes">FIPS region codes</a>
Copyright
---------
`geoip-lite` is Copyright 2011-2018 Philip Tellis <philip@bluesmoon.info> and the latest version of the code is
available at https://github.com/bluesmoon/node-geoip
License
-------
There are two licenses for the code and data. See the [LICENSE](https://github.com/bluesmoon/node-geoip/blob/master/LICENSE) file for details.

1
utils/rm/node_modules/geoip-lite/data/city.checksum generated vendored
View File

@@ -1 +0,0 @@
738794f02b34c28847582790e570b533

1
utils/rm/node_modules/geoip-lite/data/country.checksum generated vendored
View File

@@ -1 +0,0 @@
480b2d18f8423ae1f9370722fb6d12d8

BIN
utils/rm/node_modules/geoip-lite/data/geoip-city-names.dat generated vendored
View File

Binary file not shown.

BIN
utils/rm/node_modules/geoip-lite/data/geoip-city.dat generated vendored
View File

Binary file not shown.

BIN
utils/rm/node_modules/geoip-lite/data/geoip-city6.dat generated vendored
View File

Binary file not shown.

BIN
utils/rm/node_modules/geoip-lite/data/geoip-country.dat generated vendored
View File

Binary file not shown.

BIN
utils/rm/node_modules/geoip-lite/data/geoip-country6.dat generated vendored
View File

Binary file not shown.

83
utils/rm/node_modules/geoip-lite/lib/fsWatcher.js generated vendored
View File

@@ -1,83 +0,0 @@
var fs = require('fs'),
path = require('path'),
FSWatcher = {};
/**
* Takes a directory/file and watch for change. Upon change, call the
* callback.
*
* @param {String} name: name of this watcher
* @param {String} directory: path to the directory to watch
* @param {String} [filename]: (optional) specific filename to watch for,
* watches for all files in the directory if unspecified
* @param {Integer} cooldownDelay: delay to wait before triggering the callback
* @param {Function} callback: function () : called when changes are detected
**/
function makeFsWatchFilter(name, directory, filename, cooldownDelay, callback) {
var cooldownId = null;
//Delete the cooldownId and callback the outer function
function timeoutCallback() {
cooldownId = null;
callback();
}
//This function is called when there is a change in the data directory
//It sets a timer to wait for the change to be completed
function onWatchEvent(event, changedFile) {
// check to make sure changedFile is not null
if (!changedFile) {
return;
}
var filePath = path.join(directory, changedFile);
if (!filename || filename === changedFile) {
fs.exists(filePath, function onExists(exists) {
if (!exists) {
// if the changed file no longer exists, it was a deletion.
// we ignore deleted files
return;
}
//At this point, a new file system activity has been detected,
//We have to wait for file transfert to be finished before moving on.
//If a cooldownId already exists, we delete it
if (cooldownId !== null) {
clearTimeout(cooldownId);
cooldownId = null;
}
//Once the cooldownDelay has passed, the timeoutCallback function will be called
cooldownId = setTimeout(timeoutCallback, cooldownDelay);
});
}
}
//Manage the case where filename is missing (because it's optionnal)
if (typeof cooldownDelay === 'function') {
callback = cooldownDelay;
cooldownDelay = filename;
filename = null;
}
if (FSWatcher[name]) {
stopWatching(name);
}
FSWatcher[name] = fs.watch(directory, onWatchEvent);
}
/**
* Take a FSWatcher object and close it.
*
* @param {string} name: name of the watcher to close
*
**/
function stopWatching(name) {
FSWatcher[name].close();
}
module.exports.makeFsWatchFilter = makeFsWatchFilter;
module.exports.stopWatching = stopWatching;

537
utils/rm/node_modules/geoip-lite/lib/geoip.js generated vendored
View File

@@ -1,537 +0,0 @@
var fs = require('fs');
var net = require('net');
var path = require('path');
fs.existsSync = fs.existsSync || path.existsSync;
var utils = require('./utils');
var fsWatcher = require('./fsWatcher');
var async = require('async');
var watcherName = 'dataWatcher';
var geodatadir;
if (typeof global.geodatadir === 'undefined'){
geodatadir = path.join(__dirname, '/../data/');
} else {
geodatadir = global.geodatadir;
}
var dataFiles = {
city: path.join(geodatadir, 'geoip-city.dat'),
city6: path.join(geodatadir, 'geoip-city6.dat'),
cityNames: path.join(geodatadir, 'geoip-city-names.dat'),
country: path.join(geodatadir, 'geoip-country.dat'),
country6: path.join(geodatadir, 'geoip-country6.dat')
};
var privateRange4 = [
[utils.aton4('10.0.0.0'), utils.aton4('10.255.255.255')],
[utils.aton4('172.16.0.0'), utils.aton4('172.31.255.255')],
[utils.aton4('192.168.0.0'), utils.aton4('192.168.255.255')]
];
var conf4 = {
firstIP: null,
lastIP: null,
lastLine: 0,
locationBuffer: null,
locationRecordSize: 88,
mainBuffer: null,
recordSize: 24
};
var conf6 = {
firstIP: null,
lastIP: null,
lastLine: 0,
mainBuffer: null,
recordSize: 48
};
//copy original configs
var cache4 = JSON.parse(JSON.stringify(conf4));
var cache6 = JSON.parse(JSON.stringify(conf6));
var RECORD_SIZE = 10;
var RECORD_SIZE6 = 34;
function lookup4(ip) {
var fline = 0;
var floor = cache4.lastIP;
var cline = cache4.lastLine;
var ceil = cache4.firstIP;
var line;
var locId;
var buffer = cache4.mainBuffer;
var locBuffer = cache4.locationBuffer;
var privateRange = privateRange4;
var recordSize = cache4.recordSize;
var locRecordSize = cache4.locationRecordSize;
var i;
var geodata = {
range: '',
country: '',
region: '',
eu:'',
timezone:'',
city: '',
ll: [0, 0]
};
// outside IPv4 range
if (ip > cache4.lastIP || ip < cache4.firstIP) {
return null;
}
// private IP
for (i = 0; i < privateRange.length; i++) {
if (ip >= privateRange[i][0] && ip <= privateRange[i][1]) {
return null;
}
}
do {
line = Math.round((cline - fline) / 2) + fline;
floor = buffer.readUInt32BE(line * recordSize);
ceil = buffer.readUInt32BE((line * recordSize) + 4);
if (floor <= ip && ceil >= ip) {
geodata.range = [floor, ceil];
if (recordSize === RECORD_SIZE) {
geodata.country = buffer.toString('utf8', (line * recordSize) + 8, (line * recordSize) + 10);
} else {
locId = buffer.readUInt32BE((line * recordSize) + 8);
geodata.country = locBuffer.toString('utf8', (locId * locRecordSize) + 0, (locId * locRecordSize) + 2).replace(/\u0000.*/, '');
geodata.region = locBuffer.toString('utf8', (locId * locRecordSize) + 2, (locId * locRecordSize) + 5).replace(/\u0000.*/, '');
geodata.metro = locBuffer.readInt32BE((locId * locRecordSize) + 5);
geodata.ll[0] = buffer.readInt32BE((line * recordSize) + 12)/10000;//latitude
geodata.ll[1] = buffer.readInt32BE((line * recordSize) + 16)/10000; //longitude
geodata.area = buffer.readUInt32BE((line * recordSize) + 20); //longitude
geodata.eu = locBuffer.toString('utf8', (locId * locRecordSize) + 9, (locId * locRecordSize) + 10).replace(/\u0000.*/, '');
geodata.timezone = locBuffer.toString('utf8', (locId * locRecordSize) + 10, (locId * locRecordSize) + 42).replace(/\u0000.*/, '');
geodata.city = locBuffer.toString('utf8', (locId * locRecordSize) + 42, (locId * locRecordSize) + locRecordSize).replace(/\u0000.*/, '');
}
return geodata;
} else if (fline === cline) {
return null;
} else if (fline === (cline - 1)) {
if (line === fline) {
fline = cline;
} else {
cline = fline;
}
} else if (floor > ip) {
cline = line;
} else if (ceil < ip) {
fline = line;
}
} while(1);
}
function lookup6(ip) {
var buffer = cache6.mainBuffer;
var recordSize = cache6.recordSize;
var locBuffer = cache4.locationBuffer;
var locRecordSize = cache4.locationRecordSize;
var geodata = {
range: '',
country: '',
region: '',
city: '',
ll: [0, 0]
};
function readip(line, offset) {
var ii = 0;
var ip = [];
for (ii = 0; ii < 2; ii++) {
ip.push(buffer.readUInt32BE((line * recordSize) + (offset * 16) + (ii * 4)));
}
return ip;
}
cache6.lastIP = readip(cache6.lastLine, 1);
cache6.firstIP = readip(0, 0);
var fline = 0;
var floor = cache6.lastIP;
var cline = cache6.lastLine;
var ceil = cache6.firstIP;
var line;
var locId;
if (utils.cmp6(ip, cache6.lastIP) > 0 || utils.cmp6(ip, cache6.firstIP) < 0) {
return null;
}
do {
line = Math.round((cline - fline) / 2) + fline;
floor = readip(line, 0);
ceil = readip(line, 1);
if (utils.cmp6(floor, ip) <= 0 && utils.cmp6(ceil, ip) >= 0) {
if (recordSize === RECORD_SIZE6) {
geodata.country = buffer.toString('utf8', (line * recordSize) + 32, (line * recordSize) + 34).replace(/\u0000.*/, '');
} else {
locId = buffer.readUInt32BE((line * recordSize) + 32);
geodata.country = locBuffer.toString('utf8', (locId * locRecordSize) + 0, (locId * locRecordSize) + 2).replace(/\u0000.*/, '');
geodata.region = locBuffer.toString('utf8', (locId * locRecordSize) + 2, (locId * locRecordSize) + 5).replace(/\u0000.*/, '');
geodata.metro = locBuffer.readInt32BE((locId * locRecordSize) + 5);
geodata.ll[0] = buffer.readInt32BE((line * recordSize) + 36)/10000;//latitude
geodata.ll[1] = buffer.readInt32BE((line * recordSize) + 40)/10000; //longitude
geodata.area = buffer.readUInt32BE((line * recordSize) + 44); //area
geodata.eu = locBuffer.toString('utf8', (locId * locRecordSize) + 9, (locId * locRecordSize) + 10).replace(/\u0000.*/, '');
geodata.timezone = locBuffer.toString('utf8', (locId * locRecordSize) + 10, (locId * locRecordSize) + 42).replace(/\u0000.*/, '');
geodata.city = locBuffer.toString('utf8', (locId * locRecordSize) + 42, (locId * locRecordSize) + locRecordSize).replace(/\u0000.*/, '');
}
// We do not currently have detailed region/city info for IPv6, but finally have coords
return geodata;
} else if (fline === cline) {
return null;
} else if (fline === (cline - 1)) {
if (line === fline) {
fline = cline;
} else {
cline = fline;
}
} else if (utils.cmp6(floor, ip) > 0) {
cline = line;
} else if (utils.cmp6(ceil, ip) < 0) {
fline = line;
}
} while(1);
}
function get4mapped(ip) {
var ipv6 = ip.toUpperCase();
var v6prefixes = ['0:0:0:0:0:FFFF:', '::FFFF:'];
for (var i = 0; i < v6prefixes.length; i++) {
var v6prefix = v6prefixes[i];
if (ipv6.indexOf(v6prefix) == 0) {
return ipv6.substring(v6prefix.length);
}
}
return null;
}
function preload(callback) {
var datFile;
var datSize;
var asyncCache = JSON.parse(JSON.stringify(conf4));
//when the preload function receives a callback, do the task asynchronously
if (typeof arguments[0] === 'function') {
async.series([
function (cb) {
async.series([
function (cb2) {
fs.open(dataFiles.cityNames, 'r', function (err, file) {
datFile = file;
cb2(err);
});
},
function (cb2) {
fs.fstat(datFile, function (err, stats) {
datSize = stats.size;
asyncCache.locationBuffer = Buffer.alloc(datSize);
cb2(err);
});
},
function (cb2) {
fs.read(datFile, asyncCache.locationBuffer, 0, datSize, 0, cb2);
},
function (cb2) {
fs.close(datFile, cb2);
},
function (cb2) {
fs.open(dataFiles.city, 'r', function (err, file) {
datFile = file;
cb2(err);
});
},
function (cb2) {
fs.fstat(datFile, function (err, stats) {
datSize = stats.size;
cb2(err);
});
}
], function (err) {
if (err) {
if (err.code !== 'ENOENT' && err.code !== 'EBADF') {
throw err;
}
fs.open(dataFiles.country, 'r', function (err, file) {
if (err) {
cb(err);
} else {
datFile = file;
fs.fstat(datFile, function (err, stats) {
datSize = stats.size;
asyncCache.recordSize = RECORD_SIZE;
cb();
});
}
});
} else {
cb();
}
});
},
function () {
asyncCache.mainBuffer = Buffer.alloc(datSize);
async.series([
function (cb2) {
fs.read(datFile, asyncCache.mainBuffer, 0, datSize, 0, cb2);
},
function (cb2) {
fs.close(datFile, cb2);
}
], function (err) {
if (err) {
//keep old cache
} else {
asyncCache.lastLine = (datSize / asyncCache.recordSize) - 1;
asyncCache.lastIP = asyncCache.mainBuffer.readUInt32BE((asyncCache.lastLine * asyncCache.recordSize) + 4);
asyncCache.firstIP = asyncCache.mainBuffer.readUInt32BE(0);
cache4 = asyncCache;
}
callback(err);
});
}
]);
} else {
try {
datFile = fs.openSync(dataFiles.cityNames, 'r');
datSize = fs.fstatSync(datFile).size;
if (datSize === 0) {
throw {
code: 'EMPTY_FILE'
};
}
cache4.locationBuffer = Buffer.alloc(datSize);
fs.readSync(datFile, cache4.locationBuffer, 0, datSize, 0);
fs.closeSync(datFile);
datFile = fs.openSync(dataFiles.city, 'r');
datSize = fs.fstatSync(datFile).size;
} catch(err) {
if (err.code !== 'ENOENT' && err.code !== 'EBADF' && err.code !== 'EMPTY_FILE') {
throw err;
}
datFile = fs.openSync(dataFiles.country, 'r');
datSize = fs.fstatSync(datFile).size;
cache4.recordSize = RECORD_SIZE;
}
cache4.mainBuffer = Buffer.alloc(datSize);
fs.readSync(datFile, cache4.mainBuffer, 0, datSize, 0);
fs.closeSync(datFile);
cache4.lastLine = (datSize / cache4.recordSize) - 1;
cache4.lastIP = cache4.mainBuffer.readUInt32BE((cache4.lastLine * cache4.recordSize) + 4);
cache4.firstIP = cache4.mainBuffer.readUInt32BE(0);
}
}
function preload6(callback) {
var datFile;
var datSize;
var asyncCache6 = JSON.parse(JSON.stringify(conf6));
//when the preload function receives a callback, do the task asynchronously
if (typeof arguments[0] === 'function') {
async.series([
function (cb) {
async.series([
function (cb2) {
fs.open(dataFiles.city6, 'r', function (err, file) {
datFile = file;
cb2(err);
});
},
function (cb2) {
fs.fstat(datFile, function (err, stats) {
datSize = stats.size;
cb2(err);
});
}
], function (err) {
if (err) {
if (err.code !== 'ENOENT' && err.code !== 'EBADF') {
throw err;
}
fs.open(dataFiles.country6, 'r', function (err, file) {
if (err) {
cb(err);
} else {
datFile = file;
fs.fstat(datFile, function (err, stats) {
datSize = stats.size;
asyncCache6.recordSize = RECORD_SIZE6;
cb();
});
}
});
} else {
cb();
}
});
},
function () {
asyncCache6.mainBuffer = Buffer.alloc(datSize);
async.series([
function (cb2) {
fs.read(datFile, asyncCache6.mainBuffer, 0, datSize, 0, cb2);
},
function (cb2) {
fs.close(datFile, cb2);
}
], function (err) {
if (err) {
//keep old cache
} else {
asyncCache6.lastLine = (datSize / asyncCache6.recordSize) - 1;
cache6 = asyncCache6;
}
callback(err);
});
}
]);
} else {
try {
datFile = fs.openSync(dataFiles.city6, 'r');
datSize = fs.fstatSync(datFile).size;
if (datSize === 0) {
throw {
code: 'EMPTY_FILE'
};
}
} catch(err) {
if (err.code !== 'ENOENT' && err.code !== 'EBADF' && err.code !== 'EMPTY_FILE') {
throw err;
}
datFile = fs.openSync(dataFiles.country6, 'r');
datSize = fs.fstatSync(datFile).size;
cache6.recordSize = RECORD_SIZE6;
}
cache6.mainBuffer = Buffer.alloc(datSize);
fs.readSync(datFile, cache6.mainBuffer, 0, datSize, 0);
fs.closeSync(datFile);
cache6.lastLine = (datSize / cache6.recordSize) - 1;
}
}
module.exports = {
cmp: utils.cmp,
lookup: function(ip) {
if (!ip) {
return null;
} else if (typeof ip === 'number') {
return lookup4(ip);
} else if (net.isIP(ip) === 4) {
return lookup4(utils.aton4(ip));
} else if (net.isIP(ip) === 6) {
var ipv4 = get4mapped(ip);
if (ipv4) {
return lookup4(utils.aton4(ipv4));
} else {
return lookup6(utils.aton6(ip));
}
}
return null;
},
pretty: function(n) {
if (typeof n === 'string') {
return n;
} else if (typeof n === 'number') {
return utils.ntoa4(n);
} else if (n instanceof Array) {
return utils.ntoa6(n);
}
return n;
},
// Start watching for data updates. The watcher waits one minute for file transfer to
// completete before triggering the callback.
startWatchingDataUpdate: function (callback) {
fsWatcher.makeFsWatchFilter(watcherName, geodatadir, 60*1000, function () {
//Reload data
async.series([
function (cb) {
preload(cb);
},
function (cb) {
preload6(cb);
}
], callback);
});
},
// Stop watching for data updates.
stopWatchingDataUpdate: function () {
fsWatcher.stopWatching(watcherName);
},
//clear data
clear: function () {
cache4 = JSON.parse(JSON.stringify(conf4));
cache6 = JSON.parse(JSON.stringify(conf6));
},
// Reload data synchronously
reloadDataSync: function () {
preload();
preload6();
},
// Reload data asynchronously
reloadData: function (callback) {
//Reload data
async.series([
function (cb) {
preload(cb);
},
function (cb) {
preload6(cb);
}
], callback);
},
};
preload();
preload6();
//lookup4 = gen_lookup('geoip-country.dat', 4);
//lookup6 = gen_lookup('geoip-country6.dat', 16);

98
utils/rm/node_modules/geoip-lite/lib/utils.js generated vendored
View File

@@ -1,98 +0,0 @@
var utils = module.exports = {};
utils.aton4 = function(a) {
a = a.split(/\./);
return ((parseInt(a[0], 10)<<24)>>>0) + ((parseInt(a[1], 10)<<16)>>>0) + ((parseInt(a[2], 10)<<8)>>>0) + (parseInt(a[3], 10)>>>0);
};
utils.aton6 = function(a) {
a = a.replace(/"/g, '').split(/:/);
var l = a.length - 1;
var i;
if (a[l] === '') {
a[l] = 0;
}
if (l < 7) {
a.length = 8;
for (i = l; i >= 0 && a[i] !== ''; i--) {
a[7-l+i] = a[i];
}
}
for (i = 0; i < 8; i++) {
if (!a[i]) {
a[i]=0;
} else {
a[i] = parseInt(a[i], 16);
}
}
var r = [];
for (i = 0; i<4; i++) {
r.push(((a[2*i]<<16) + a[2*i+1])>>>0);
}
return r;
};
utils.cmp = function(a, b) {
if (typeof a === 'number' && typeof b === 'number') {
return (a < b ? -1 : (a > b ? 1 : 0));
}
if (a instanceof Array && b instanceof Array) {
return this.cmp6(a, b);
}
return null;
};
utils.cmp6 = function(a, b) {
for (var ii = 0; ii < 2; ii++) {
if (a[ii] < b[ii]) {
return -1;
}
if (a[ii] > b[ii]) {
return 1;
}
}
return 0;
};
utils.isPrivateIP = function(addr) {
addr = addr.toString();
return addr.match(/^10\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})/) != null ||
addr.match(/^192\.168\.([0-9]{1,3})\.([0-9]{1,3})/) != null ||
addr.match(/^172\.16\.([0-9]{1,3})\.([0-9]{1,3})/) != null ||
addr.match(/^127\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})/) != null ||
addr.match(/^169\.254\.([0-9]{1,3})\.([0-9]{1,3})/) != null ||
addr.match(/^fc00:/) != null || addr.match(/^fe80:/) != null;
};
utils.ntoa4 = function(n) {
n = n.toString();
n = '' + (n>>>24&0xff) + '.' + (n>>>16&0xff) + '.' + (n>>>8&0xff) + '.' + (n&0xff);
return n;
};
utils.ntoa6 = function(n) {
var a = "[";
for (var i = 0; i<n.length; i++) {
a += (n[i]>>>16).toString(16) + ':';
a += (n[i]&0xffff).toString(16) + ':';
}
a = a.replace(/:$/, ']').replace(/:0+/g, ':').replace(/::+/, '::');
return a;
};

36
utils/rm/node_modules/geoip-lite/package.json generated vendored
View File

@@ -1,36 +0,0 @@
{
"name" : "geoip-lite",
"version" : "1.4.5",
"description" : "A light weight native JavaScript implementation of GeoIP API from MaxMind",
"keywords" : ["geo", "geoip", "ip", "ipv4", "ipv6", "geolookup", "maxmind", "geolite"],
"homepage" : "https://github.com/geoip-lite/node-geoip",
"author" : "Philip Tellis <philip@bluesmoon.info> (http://bluesmoon.info/)",
"files" : ["lib/", "data/", "test/","scripts/"],
"main" : "lib/geoip.js",
"repository" : { "type": "git", "url": "git://github.com/geoip-lite/node-geoip.git" },
"engines" : { "node": ">=5.10.0" },
"scripts": {
"pretest": "eslint .",
"test": "nodeunit --reporter=minimal test/tests.js",
"updatedb": "node scripts/updatedb.js",
"updatedb-debug": "node scripts/updatedb.js debug",
"updatedb-force": "node scripts/updatedb.js force"
},
"dependencies": {
"async": "2.1 - 2.6.4",
"chalk": "4.1 - 4.1.2",
"iconv-lite": "0.4.13 - 0.6.3",
"ip-address": "5.8.9 - 5.9.4",
"lazy": "1.0.11",
"rimraf": "2.5.2 - 2.7.1",
"yauzl": "2.9.2 - 2.10.0"
},
"config": {
"update": true
},
"devDependencies": {
"eslint": "^5.12.1",
"nodeunit": "^0.11.2"
},
"license": "Apache-2.0"
}

647
utils/rm/node_modules/geoip-lite/scripts/updatedb.js generated vendored
View File

@@ -1,647 +0,0 @@
// fetches and converts maxmind lite databases
'use strict';
var user_agent = 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.36 Safari/537.36';
var fs = require('fs');
var http = require('http');
var https = require('https');
var path = require('path');
var url = require('url');
var zlib = require('zlib');
fs.existsSync = fs.existsSync || path.existsSync;
var async = require('async');
var chalk = require('chalk');
var iconv = require('iconv-lite');
var lazy = require('lazy');
var rimraf = require('rimraf').sync;
var yauzl = require('yauzl');
var utils = require('../lib/utils');
var Address6 = require('ip-address').Address6;
var Address4 = require('ip-address').Address4;
var args = process.argv.slice(2);
var license_key = args.find(function(arg) {
return arg.match(/^license_key=[a-zA-Z0-9]+/) !== null;
});
if (typeof license_key === 'undefined' && typeof process.env.LICENSE_KEY !== 'undefined') {
license_key = 'license_key='+process.env.LICENSE_KEY;
}
var dataPath = path.join(__dirname, '..', 'data');
var tmpPath = path.join(__dirname, '..', 'tmp');
var countryLookup = {};
var cityLookup = {};
var databases = [
{
type: 'country',
url: 'https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&suffix=zip&'+license_key,
checksum: 'https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-Country-CSV&suffix=zip.sha256&'+license_key,
fileName: 'GeoLite2-Country-CSV.zip',
src: [
'GeoLite2-Country-Locations-en.csv',
'GeoLite2-Country-Blocks-IPv4.csv',
'GeoLite2-Country-Blocks-IPv6.csv'
],
dest: [
'',
'geoip-country.dat',
'geoip-country6.dat'
]
},
{
type: 'city',
url: 'https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City-CSV&suffix=zip&'+license_key,
checksum: 'https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City-CSV&suffix=zip.sha256&'+license_key,
fileName: 'GeoLite2-City-CSV.zip',
src: [
'GeoLite2-City-Locations-en.csv',
'GeoLite2-City-Blocks-IPv4.csv',
'GeoLite2-City-Blocks-IPv6.csv'
],
dest: [
'geoip-city-names.dat',
'geoip-city.dat',
'geoip-city6.dat'
]
}
];
function mkdir(name) {
var dir = path.dirname(name);
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir);
}
}
// Ref: http://stackoverflow.com/questions/8493195/how-can-i-parse-a-csv-string-with-javascript
// Return array of string values, or NULL if CSV string not well formed.
// Return array of string values, or NULL if CSV string not well formed.
function try_fixing_line(line) {
var pos1 = 0;
var pos2 = -1;
// escape quotes
line = line.replace(/""/,'\\"').replace(/'/g,"\\'");
while(pos1 < line.length && pos2 < line.length) {
pos1 = pos2;
pos2 = line.indexOf(',', pos1 + 1);
if(pos2 < 0) pos2 = line.length;
if(line.indexOf("'", (pos1 || 0)) > -1 && line.indexOf("'", pos1) < pos2 && line[pos1 + 1] != '"' && line[pos2 - 1] != '"') {
line = line.substr(0, pos1 + 1) + '"' + line.substr(pos1 + 1, pos2 - pos1 - 1) + '"' + line.substr(pos2, line.length - pos2);
pos2 = line.indexOf(',', pos2 + 1);
if(pos2 < 0) pos2 = line.length;
}
}
return line;
}
function CSVtoArray(text) {
var re_valid = /^\s*(?:'[^'\\]*(?:\\[\S\s][^'\\]*)*'|"[^"\\]*(?:\\[\S\s][^"\\]*)*"|[^,'"\s\\]*(?:\s+[^,'"\s\\]+)*)\s*(?:,\s*(?:'[^'\\]*(?:\\[\S\s][^'\\]*)*'|"[^"\\]*(?:\\[\S\s][^"\\]*)*"|[^,'"\s\\]*(?:\s+[^,'"\s\\]+)*)\s*)*$/;
var re_value = /(?!\s*$)\s*(?:'([^'\\]*(?:\\[\S\s][^'\\]*)*)'|"([^"\\]*(?:\\[\S\s][^"\\]*)*)"|([^,'"\s\\]*(?:\s+[^,'"\s\\]+)*))\s*(?:,|$)/g;
// Return NULL if input string is not well formed CSV string.
if (!re_valid.test(text)){
text = try_fixing_line(text);
if(!re_valid.test(text))
return null;
}
var a = []; // Initialize array to receive values.
text.replace(re_value, // "Walk" the string using replace with callback.
function(m0, m1, m2, m3) {
// Remove backslash from \' in single quoted values.
if (m1 !== undefined) a.push(m1.replace(/\\'/g, "'"));
// Remove backslash from \" in double quoted values.
else if (m2 !== undefined) a.push(m2.replace(/\\"/g, '"').replace(/\\'/g, "'"));
else if (m3 !== undefined) a.push(m3);
return ''; // Return empty string.
});
// Handle special case of empty last value.
if (/,\s*$/.test(text)) a.push('');
return a;
}
function getHTTPOptions(downloadUrl) {
var options = url.parse(downloadUrl);
options.headers = {
'User-Agent': user_agent
};
if (process.env.http_proxy || process.env.https_proxy) {
try {
var HttpsProxyAgent = require('https-proxy-agent');
options.agent = new HttpsProxyAgent(process.env.http_proxy || process.env.https_proxy);
}
catch (e) {
console.error("Install https-proxy-agent to use an HTTP/HTTPS proxy");
process.exit(-1);
}
}
return options;
}
function check(database, cb) {
if (args.indexOf("force") !== -1) {
//we are forcing database upgrade,
//so not even using checksums
return cb(null, database);
}
var checksumUrl = database.checksum;
if (typeof checksumUrl === "undefined") {
//no checksum url to check, skipping
return cb(null, database);
}
//read existing checksum file
fs.readFile(path.join(dataPath, database.type+".checksum"), {encoding: 'utf8'}, function(err, data) {
if (!err && data && data.length) {
database.checkValue = data;
}
console.log('Checking ', database.fileName);
function onResponse(response) {
var status = response.statusCode;
if (status !== 200) {
console.log(chalk.red('ERROR') + ': HTTP Request Failed [%d %s]', status, http.STATUS_CODES[status]);
client.abort();
process.exit();
}
var str = "";
response.on("data", function (chunk) {
str += chunk;
});
response.on("end", function () {
if (str && str.length) {
if (str == database.checkValue) {
console.log(chalk.green('Database "' + database.type + '" is up to date'));
database.skip = true;
}
else {
console.log(chalk.green('Database ' + database.type + ' has new data'));
database.checkValue = str;
}
}
else {
console.log(chalk.red('ERROR') + ': Could not retrieve checksum for', database.type, chalk.red('Aborting'));
console.log('Run with "force" to update without checksum');
client.abort();
process.exit();
}
cb(null, database);
});
}
var client = https.get(getHTTPOptions(checksumUrl), onResponse);
});
}
function fetch(database, cb) {
if (database.skip) {
return cb(null, null, null, database);
}
var downloadUrl = database.url;
var fileName = database.fileName;
var gzip = path.extname(fileName) === '.gz';
if (gzip) {
fileName = fileName.replace('.gz', '');
}
var tmpFile = path.join(tmpPath, fileName);
if (fs.existsSync(tmpFile)) {
return cb(null, tmpFile, fileName, database);
}
console.log('Fetching ', fileName);
function onResponse(response) {
var status = response.statusCode;
if (status !== 200) {
console.log(chalk.red('ERROR') + ': HTTP Request Failed [%d %s]', status, http.STATUS_CODES[status]);
client.abort();
process.exit();
}
var tmpFilePipe;
var tmpFileStream = fs.createWriteStream(tmpFile);
if (gzip) {
tmpFilePipe = response.pipe(zlib.createGunzip()).pipe(tmpFileStream);
} else {
tmpFilePipe = response.pipe(tmpFileStream);
}
tmpFilePipe.on('close', function() {
console.log(chalk.green(' DONE'));
cb(null, tmpFile, fileName, database);
});
}
mkdir(tmpFile);
var client = https.get(getHTTPOptions(downloadUrl), onResponse);
process.stdout.write('Retrieving ' + fileName + ' ...');
}
function extract(tmpFile, tmpFileName, database, cb) {
if (database.skip) {
return cb(null, database);
}
if (path.extname(tmpFileName) !== '.zip') {
cb(null, database);
} else {
process.stdout.write('Extracting ' + tmpFileName + ' ...');
yauzl.open(tmpFile, {autoClose: true, lazyEntries: true}, function(err, zipfile) {
if (err) {
throw err;
}
zipfile.readEntry();
zipfile.on("entry", function(entry) {
if (/\/$/.test(entry.fileName)) {
// Directory file names end with '/'.
// Note that entries for directories themselves are optional.
// An entry's fileName implicitly requires its parent directories to exist.
zipfile.readEntry();
} else {
// file entry
zipfile.openReadStream(entry, function(err, readStream) {
if (err) {
throw err;
}
readStream.on("end", function() {
zipfile.readEntry();
});
var filePath = entry.fileName.split("/");
// filePath will always have length >= 1, as split() always returns an array of at least one string
var fileName = filePath[filePath.length - 1];
readStream.pipe(fs.createWriteStream(path.join(tmpPath, fileName)));
});
}
});
zipfile.once("end", function() {
console.log(chalk.green(' DONE'));
cb(null, database);
});
});
}
}
function processLookupCountry(src, cb){
function processLine(line) {
var fields = CSVtoArray(line);
if (!fields || fields.length < 6) {
console.log("weird line: %s::", line);
return;
}
countryLookup[fields[0]] = fields[4];
}
var tmpDataFile = path.join(tmpPath, src);
process.stdout.write('Processing Lookup Data (may take a moment) ...');
lazy(fs.createReadStream(tmpDataFile))
.lines
.map(function(byteArray) {
return iconv.decode(byteArray, 'latin1');
})
.skip(1)
.map(processLine)
.on('pipe', function() {
console.log(chalk.green(' DONE'));
cb();
});
}
function processCountryData(src, dest, cb) {
var lines=0;
function processLine(line) {
var fields = CSVtoArray(line);
if (!fields || fields.length < 6) {
console.log("weird line: %s::", line);
return;
}
lines++;
var sip;
var eip;
var rngip;
var cc = countryLookup[fields[1]];
var b;
var bsz;
var i;
if(cc){
if (fields[0].match(/:/)) {
// IPv6
bsz = 34;
rngip = new Address6(fields[0]);
sip = utils.aton6(rngip.startAddress().correctForm());
eip = utils.aton6(rngip.endAddress().correctForm());
b = Buffer.alloc(bsz);
for (i = 0; i < sip.length; i++) {
b.writeUInt32BE(sip[i], i * 4);
}
for (i = 0; i < eip.length; i++) {
b.writeUInt32BE(eip[i], 16 + (i * 4));
}
} else {
// IPv4
bsz = 10;
rngip = new Address4(fields[0]);
sip = parseInt(rngip.startAddress().bigInteger(),10);
eip = parseInt(rngip.endAddress().bigInteger(),10);
b = Buffer.alloc(bsz);
b.fill(0);
b.writeUInt32BE(sip, 0);
b.writeUInt32BE(eip, 4);
}
b.write(cc, bsz - 2);
fs.writeSync(datFile, b, 0, bsz, null);
if(Date.now() - tstart > 5000) {
tstart = Date.now();
process.stdout.write('\nStill working (' + lines + ') ...');
}
}
}
var dataFile = path.join(dataPath, dest);
var tmpDataFile = path.join(tmpPath, src);
rimraf(dataFile);
mkdir(dataFile);
process.stdout.write('Processing Data (may take a moment) ...');
var tstart = Date.now();
var datFile = fs.openSync(dataFile, "w");
lazy(fs.createReadStream(tmpDataFile))
.lines
.map(function(byteArray) {
return iconv.decode(byteArray, 'latin1');
})
.skip(1)
.map(processLine)
.on('pipe', function() {
console.log(chalk.green(' DONE'));
cb();
});
}
function processCityData(src, dest, cb) {
var lines = 0;
function processLine(line) {
if (line.match(/^Copyright/) || !line.match(/\d/)) {
return;
}
var fields = CSVtoArray(line);
if (!fields) {
console.log("weird line: %s::", line);
return;
}
var sip;
var eip;
var rngip;
var locId;
var b;
var bsz;
var lat;
var lon;
var area;
var i;
lines++;
if (fields[0].match(/:/)) {
// IPv6
var offset = 0;
bsz = 48;
rngip = new Address6(fields[0]);
sip = utils.aton6(rngip.startAddress().correctForm());
eip = utils.aton6(rngip.endAddress().correctForm());
locId = parseInt(fields[1], 10);
locId = cityLookup[locId];
b = Buffer.alloc(bsz);
b.fill(0);
for (i = 0; i < sip.length; i++) {
b.writeUInt32BE(sip[i], offset);
offset += 4;
}
for (i = 0; i < eip.length; i++) {
b.writeUInt32BE(eip[i], offset);
offset += 4;
}
b.writeUInt32BE(locId>>>0, 32);
lat = Math.round(parseFloat(fields[7]) * 10000);
lon = Math.round(parseFloat(fields[8]) * 10000);
area = parseInt(fields[9], 10);
b.writeInt32BE(lat,36);
b.writeInt32BE(lon,40);
b.writeInt32BE(area,44);
} else {
// IPv4
bsz = 24;
rngip = new Address4(fields[0]);
sip = parseInt(rngip.startAddress().bigInteger(),10);
eip = parseInt(rngip.endAddress().bigInteger(),10);
locId = parseInt(fields[1], 10);
locId = cityLookup[locId];
b = Buffer.alloc(bsz);
b.fill(0);
b.writeUInt32BE(sip>>>0, 0);
b.writeUInt32BE(eip>>>0, 4);
b.writeUInt32BE(locId>>>0, 8);
lat = Math.round(parseFloat(fields[7]) * 10000);
lon = Math.round(parseFloat(fields[8]) * 10000);
area = parseInt(fields[9], 10);
b.writeInt32BE(lat,12);
b.writeInt32BE(lon,16);
b.writeInt32BE(area,20);
}
fs.writeSync(datFile, b, 0, b.length, null);
if(Date.now() - tstart > 5000) {
tstart = Date.now();
process.stdout.write('\nStill working (' + lines + ') ...');
}
}
var dataFile = path.join(dataPath, dest);
var tmpDataFile = path.join(tmpPath, src);
rimraf(dataFile);
process.stdout.write('Processing Data (may take a moment) ...');
var tstart = Date.now();
var datFile = fs.openSync(dataFile, "w");
lazy(fs.createReadStream(tmpDataFile))
.lines
.map(function(byteArray) {
return iconv.decode(byteArray, 'latin1');
})
.skip(1)
.map(processLine)
.on('pipe', cb);
}
function processCityDataNames(src, dest, cb) {
var locId = null;
var linesCount = 0;
function processLine(line) {
if (line.match(/^Copyright/) || !line.match(/\d/)) {
return;
}
var b;
var sz = 88;
var fields = CSVtoArray(line);
if (!fields) {
//lot's of cities contain ` or ' in the name and can't be parsed correctly with current method
console.log("weird line: %s::", line);
return;
}
locId = parseInt(fields[0]);
cityLookup[locId] = linesCount;
var cc = fields[4];
var rg = fields[6];
var city = fields[10];
var metro = parseInt(fields[11]);
//other possible fields to include
var tz = fields[12];
var eu = fields[13];
b = Buffer.alloc(sz);
b.fill(0);
b.write(cc, 0);//country code
b.write(rg, 2);//region
if(metro) {
b.writeInt32BE(metro, 5);
}
b.write(eu,9);//is in eu
b.write(tz,10);//timezone
b.write(city, 42);//cityname
fs.writeSync(datFile, b, 0, b.length, null);
linesCount++;
}
var dataFile = path.join(dataPath, dest);
var tmpDataFile = path.join(tmpPath, src);
rimraf(dataFile);
var datFile = fs.openSync(dataFile, "w");
lazy(fs.createReadStream(tmpDataFile))
.lines
.map(function(byteArray) {
return iconv.decode(byteArray, 'utf-8');
})
.skip(1)
.map(processLine)
.on('pipe', cb);
}
function processData(database, cb) {
if (database.skip) {
return cb(null, database);
}
var type = database.type;
var src = database.src;
var dest = database.dest;
if (type === 'country') {
if(Array.isArray(src)){
processLookupCountry(src[0], function() {
processCountryData(src[1], dest[1], function() {
processCountryData(src[2], dest[2], function() {
cb(null, database);
});
});
});
}
else{
processCountryData(src, dest, function() {
cb(null, database);
});
}
} else if (type === 'city') {
processCityDataNames(src[0], dest[0], function() {
processCityData(src[1], dest[1], function() {
console.log("city data processed");
processCityData(src[2], dest[2], function() {
console.log(chalk.green(' DONE'));
cb(null, database);
});
});
});
}
}
function updateChecksum(database, cb) {
if (database.skip || !database.checkValue) {
//don't need to update checksums cause it was not fetched or did not change
return cb();
}
fs.writeFile(path.join(dataPath, database.type+".checksum"), database.checkValue, 'utf8', function(err){
if (err) console.log(chalk.red('Failed to Update checksums.'), "Database:", database.type);
cb();
});
}
if (!license_key) {
console.log(chalk.red('ERROR') + ': Missing license_key');
process.exit(1);
}
rimraf(tmpPath);
mkdir(tmpPath);
async.eachSeries(databases, function(database, nextDatabase) {
async.seq(check, fetch, extract, processData, updateChecksum)(database, nextDatabase);
}, function(err) {
if (err) {
console.log(chalk.red('Failed to Update Databases from MaxMind.'), err);
process.exit(1);
} else {
console.log(chalk.green('Successfully Updated Databases from MaxMind.'));
if (args.indexOf("debug") !== -1) console.log(chalk.yellow.bold('Notice: temporary files are not deleted for debug purposes.'));
else rimraf(tmpPath);
process.exit(0);
}
});

56
utils/rm/node_modules/geoip-lite/test/geo-lookup.js generated vendored
View File

@@ -1,56 +0,0 @@
var assert = require('assert');
var t1 =+ new Date();
var geoip = require('../lib/geoip');
var t2 =+ new Date();
if (process.argv.length > 2) {
console.dir(geoip.lookup(process.argv[2]));
var t3 =+ new Date();
console.log('Startup: %dms, exec: %dms', t2 - t1, t3 - t2);
process.exit();
}
var f = [];
var ip;
var n = 30000;
var nf = [];
var r;
var ts =+ new Date();
for (var i = 0; i < n; i++) {
if ((i % 2) === 0) {
ip = Math.round((Math.random() * 0xff000000)+ 0xffffff);
} else {
ip = '2001:' +
Math.round(Math.random()*0xffff).toString(16) + ':' +
Math.round(Math.random()*0xffff).toString(16) + ':' +
Math.round(Math.random()*0xffff).toString(16) + ':' +
Math.round(Math.random()*0xffff).toString(16) + ':' +
Math.round(Math.random()*0xffff).toString(16) + ':' +
Math.round(Math.random()*0xffff).toString(16) + ':' +
Math.round(Math.random()*0xffff).toString(16) + '';
}
r = geoip.lookup(ip);
if (r === null) {
nf.push(ip);
continue;
}
f.push([ip, r]);
assert.ok(geoip.cmp(ip, r.range[0]) >= 0 , 'Problem with ' + geoip.pretty(ip) + ' < ' + geoip.pretty(r.range[0]));
assert.ok(geoip.cmp(ip, r.range[1]) <= 0 , 'Problem with ' + geoip.pretty(ip) + ' > ' + geoip.pretty(r.range[1]));
}
var te =+ new Date();
/*
f.forEach(function(ip) {
console.log("%s bw %s & %s is %s", geoip.pretty(ip[0]), geoip.pretty(ip[1].range[0]), geoip.pretty(ip[1].range[1]), ip[1].country);
});
*/
console.log("Found %d (%d/%d) ips in %dms (%s ip/s) (%sμs/ip)", n, f.length, nf.length, te-ts, (n*1000 / (te-ts)).toFixed(3), ((te-ts) * 1000 / n).toFixed(0));
console.log("Took %d ms to startup", t2 - t1);

3
utils/rm/node_modules/geoip-lite/test/memory_usage.js generated vendored
View File

@@ -1,3 +0,0 @@
// eslint-disable-next-line no-unused-vars
var geoip = require('../lib/geoip');
console.log(process.memoryUsage());

172
utils/rm/node_modules/geoip-lite/test/tests.js generated vendored
View File

@@ -1,172 +0,0 @@
var geoip = require('../lib/geoip');
module.exports = {
testLookup: function(test) {
test.expect(2);
var ip = '8.8.4.4';
var ipv6 = '2001:4860:b002::68';
var actual = geoip.lookup(ip);
test.ok(actual, 'should return data about IPv4.');
actual = geoip.lookup(ipv6);
test.ok(actual, 'should return data about IPv6.');
test.done();
},
testDataIP4: function(test) {
test.expect(9);
var ip = '72.229.28.185';
var actual = geoip.lookup(ip);
test.notStrictEqual(actual.range, undefined, 'should contain IPv4 range');
test.strictEqual(actual.country, 'US', "should match country");
test.strictEqual(actual.region, 'NY', "should match region");
test.strictEqual(actual.eu, '0', "should match eu");
test.strictEqual(actual.timezone, 'America/New_York', "should match timezone");
test.strictEqual(actual.city, 'New York', "should match city");
test.ok(actual.ll, 'should contain coordinates');
test.strictEqual(actual.metro, 501, "should match metro");
test.strictEqual(actual.area, 1, "should match area");
test.done();
},
testDataIP6: function(test) {
test.expect(9);
var ipv6 = '2001:1c04:400::1';
var actual = geoip.lookup(ipv6);
test.notStrictEqual(actual.range, undefined, 'should contain IPv6 range');
test.strictEqual(actual.country, 'NL', "should match country");
test.strictEqual(actual.region, 'NH', "should match region");
test.strictEqual(actual.eu, '1', "should match eu");
test.strictEqual(actual.timezone, 'Europe/Amsterdam', "should match timezone");
test.strictEqual(actual.city, 'Amsterdam', "should match city");
test.ok(actual.ll, 'should contain coordinates');
test.strictEqual(actual.metro, 0, "should match metro");
test.strictEqual(actual.area, 5, "should match area");
test.done();
},
testUTF8: function(test) {
test.expect(2);
var ip = "2.139.175.1";
var expected = "Pamplona";
var actual = geoip.lookup(ip);
test.ok(actual, "Should return a non-null value for " + ip);
test.equal(actual.city, expected, "UTF8 city name does not match");
test.done();
},
testMetro: function(test) {
test.expect(2);
var actual = geoip.lookup("23.240.63.68");
test.equal(actual.city, "Riverside"); //keeps changing with each update from one city to other (close to each other geographically)
test.equal(actual.metro, 803);
test.done();
},
testIPv4MappedIPv6: function (test) {
test.expect(2);
var actual = geoip.lookup("195.16.170.74");
test.equal(actual.city, "");
test.equal(actual.metro, 0);
test.done();
},
testSyncReload: function (test) {
test.expect(6);
//get original data
var before4 = geoip.lookup("75.82.117.180");
test.notEqual(before4, null);
var before6 = geoip.lookup("::ffff:173.185.182.82");
test.notEqual(before6, null);
//clear data;
geoip.clear();
//make sure data is cleared
var none4 = geoip.lookup("75.82.117.180");
test.equal(none4, null);
var none6 = geoip.lookup("::ffff:173.185.182.82");
test.equal(none6, null);
//reload data synchronized
geoip.reloadDataSync();
//make sure we have value from before
var after4 = geoip.lookup("75.82.117.180");
test.deepEqual(before4, after4);
var after6 = geoip.lookup("::ffff:173.185.182.82");
test.deepEqual(before6, after6);
test.done();
},
testAsyncReload: function (test) {
test.expect(6);
//get original data
var before4 = geoip.lookup("75.82.117.180");
test.notEqual(before4, null);
var before6 = geoip.lookup("::ffff:173.185.182.82");
test.notEqual(before6, null);
//clear data;
geoip.clear();
//make sure data is cleared
var none4 = geoip.lookup("75.82.117.180");
test.equal(none4, null);
var none6 = geoip.lookup("::ffff:173.185.182.82");
test.equal(none6, null);
//reload data asynchronously
geoip.reloadData(function(){
//make sure we have value from before
var after4 = geoip.lookup("75.82.117.180");
test.deepEqual(before4, after4);
var after6 = geoip.lookup("::ffff:173.185.182.82");
test.deepEqual(before6, after6);
test.done();
});
}
};