索引更多数据

.gitignore

@@ -10,6 +10,7 @@ sphinx.conf
 *.p2p
 downloads.json
 node.exe
+.idea
 
 /dist
 /temp

Dockerfile

@@ -3,7 +3,6 @@ ARG DEBIAN_FRONTEND=noninteractive
 RUN mkdir -p /home/node/app/node_modules && chown -R node:node /home/node/app
 WORKDIR /home/node/app
 COPY --chown=node:node . .
-RUN npm install -g npm
 USER node
 
 RUN npm install --force
@@ -11,4 +10,4 @@ RUN ls -la
 RUN npm run buildweb
 
 EXPOSE 8095
-CMD [ "node", "src/background/server.js" ]
+CMD [ "node", "src/background/server.js" ]

README.md

@@ -30,6 +30,7 @@ BitTorrent search program for desktop and web. Collect and navigate over base of
 * Drag and drop torrents (expand local search database with specific torrents)
 * Descriptions association from trackers
 * Torrent generation and automatic adding to search DB
+* [WebSockets & REST API for server/search engine. You can made search request and create your own UI client.](docs/API.md)
 
 ## Architecture
 ![Basic Architecture](docs/img/ratsarch.png)
@@ -91,6 +92,8 @@ Now you can get access to web interface on 8095 port: http://localhost:8095
 
 [More about server compatibility and known issues](docs/SERVER_COMPATIBILITY.md)
 
+[API usage implementation for clients](docs/API.md)
+
 ## Docker image
 
 You can simply run docker image with prepared server version. Just download last sources:

app/donate.html

@@ -63,21 +63,28 @@
                     C243.779,80.572,238.768,71.728,220.195,71.427z"/>
   <center>
 
-  <div class='patreon' style="display: flex;">
-  <div class="widget">
-  <a href="https://opencollective.com/RatsSearch">
+  <div class='patreon' style="display: flex; flex-wrap: wrap; justify-content: center;">
+    <div class="widget">
+    <a href="https://github.com/sponsors/DEgITx">
 
-  Subscribe on Open Collective (continuous support development with 1$ or more)
-  <img src="open-collective.png" style="width: 400px; padding-top: 12px; opacity: 0.9;" />
+    Support development directly via GitHub sponsorship  (recomended)
+    <img src="github.jpg" style="width: 400px; padding-top: 12px; opacity: 0.9;" />
 
-  </a>
-  </div>
-  <iframe src="https://money.yandex.ru/quickpay/shop-widget?writer=seller&targets=Rats%20Search&targets-hint=&default-sum=500&button-text=11&payment-type-choice=on&mobile-payment-type-choice=on&hint=&successURL=&quickpay=shop&account=410012059502693" width="450" height="213" frameborder="0" allowtransparency="true" scrolling="no"></iframe>
+    </a>
+    </div>
+    <div class="widget">
+    <a href="https://opencollective.com/RatsSearch">
+
+    Subscribe on Open Collective (continuous support development with 1$ or more)
+    <img src="open-collective.png" style="width: 400px; padding-top: 12px; opacity: 0.9;" />
+
+    </a>
+    </div>
   </div>
 
   <br />
-  <div style='display: flex; align-items: center; width: 100%; justify-content: center;'><img style="height: 25px; width: 25px; padding: 5px;" src="https://png.icons8.com/ios/1600/webmoney.png" /> WMR (Webmoney):&nbsp; <b>R227938595852</b></div>
-  <div style='display: flex; align-items: center; width: 100%; justify-content: center;'><img style="height: 25px; width: 25px; padding: 5px;" src="https://png.icons8.com/ios/1600/webmoney.png" /> WMZ (Webmoney):&nbsp; <b>Z133588309220</b></div>
+  <!-- <div style='display: flex; align-items: center; width: 100%; justify-content: center;'><img style="height: 25px; width: 25px; padding: 5px;" src="https://png.icons8.com/ios/1600/webmoney.png" /> WMR (Webmoney):&nbsp; <b>R227938595852</b></div>
+  <div style='display: flex; align-items: center; width: 100%; justify-content: center;'><img style="height: 25px; width: 25px; padding: 5px;" src="https://png.icons8.com/ios/1600/webmoney.png" /> WMZ (Webmoney):&nbsp; <b>Z133588309220</b></div> -->
   </center>
 </body>
 </html>

docs/API.md

@@ -0,0 +1,125 @@
+# Basic description
+
+Rats Search server using WebSockets for internal communication. It require two way communication and using socket.io internally. As alternative added REST API implementation, but because of one-way limitation, it require periodic check of backward messages with **/api/queue** . You free to choise one of the ways for communication and implementation of your own client.
+
+## WebSockets communication (1-way)
+
+
+## REST API communication (2-way)
+
+### Installation
+
+At first you need to prepare server non-interface version
+
+```bash
+git clone --recurse-submodules https://github.com/DEgITx/rats-search.git
+npm install --force
+npm run buildweb
+npm run server
+```
+
+### Configuration
+
+You need to enable REST API configuration (disabled by default):
+
+edit **rats.json** (change only restApi value):
+```json
+{
+    "restApi": true
+}
+```
+
+set restApi to true, save
+
+### API usage
+
+#### Search of torrents
+
+endpoint (GET REQUEST):
+```
+https://localhost:8095/api/searchTorrent?text=DEgITx&navigation={}
+```
+
+example of request:
+```json
+{
+    "text": "Search Name of the torrent",
+    "navigation": {
+        "index": 0,
+        "limit": 10,
+        "orderBy": "order_field",
+        "orderDesc": "DESC",
+        "safeSearch": false
+    }
+}
+```
+
+| Field | Type | Optional | Default Value | Description |
+| ----- | ---- | -------- | ------------- | ----------- |
+| text | string | ❎ |  | torrent search name |
+| navigation | object (Navigation) | ✅ |  | object with navigation params |
+|       index | int | ✅ | 0 | stating of torrent index of navigation |
+|       limit | int | ✅ | 10 | max number of results on page |
+|       orderBy | text | ✅ |  | field which is using for order results |
+|       orderDesc | enum [**DESC, ASC**] | ✅ | ASC | sort direction of the field |
+|       safeSearch | bool | ✅ | true | disable/enable safe search for torrents |
+|       type | string | ✅ |  | type of content for search |
+|       size | object (Interval) | ✅ |  | size of torrent |
+|            min | uint64 | ✅ |  | minumum size of the torrent |
+|            max | uint64 | ✅ |  | maximum size of the torrent |
+|       files | object (Interval) | ✅ |  | files on the torrent |
+|            min | int | ✅ |  | minumum size of the torrent |
+|            max | int | ✅ |  | maximum size of the torrent |
+
+### Reading queue
+
+As said before after each request and periodicly you need to read queue for additional messages.
+
+endpoint (GET REQUEST):
+```
+https://localhost:8095/api/queue
+```
+
+after executing of search **/api/searchTorrent** request **additional result of search will be in queue**!
+
+### Search of the torrent by files
+
+endpoint (GET REQUEST):
+```
+https://localhost:8095/api/searchFiles?text=TorrentWithFileName&navigation={}
+```
+
+| Field | Type | Optional | Default Value | Description |
+| ----- | ---- | -------- | ------------- | ----------- |
+| text | string | ❎ |  | torrent search name |
+| navigation | object (Navigation) | ✅ |  | object with navigation params |
+|       index | int | ✅ | 0 | stating of torrent index of navigation |
+|       limit | int | ✅ | 10 | max number of results on page |
+|       orderBy | text | ✅ |  | field which is using for order results |
+|       orderDesc | enum [**DESC, ASC**] | ✅ | ASC | sort direction of the field |
+|       safeSearch | bool | ✅ | true | disable/enable safe search for torrents |
+
+### Recheck trackers info for the torrent
+
+endpoint (GET REQUEST):
+```
+https://localhost:8095/api/checkTrackers?hash=29ebe63feb8be91b6dcff02bacc562d9a99ea864
+```
+
+| Field | Type | Optional | Default Value | Description |
+| ----- | ---- | -------- | ------------- | ----------- |
+| hash | string | ❎ |  | torrent hash to refresh token |
+
+### Top torrents
+
+endpoint (GET REQUEST):
+```
+https://localhost:8095/api/topTorrents?type=video&navigation={"time":"week"}
+```
+
+| Field | Type | Optional | Default Value | Description |
+| ----- | ---- | -------- | ------------- | ----------- |
+| type | string | ❎ |  | type of category for top of the torrents |
+| navigation | object (Navigation) | ✅ |  | object with navigation params (check /api/searchTorrent for mo details) |
+|       time | enum [hours, week, month] | ✅ |  | time for the top
+

package.json

@@ -109,7 +109,7 @@
     "buildweb": "node src/background/webpack.js"
   },
   "dependencies": {
-    "@electron/remote": "^2.0.8",
+    "@electron/remote": "^2.0.11",
     "ansi-256-colors": "^1.1.0",
     "bencode": "2.0.1",
     "bitfield": "3.0.0",
@@ -120,7 +120,7 @@
     "detect-onebyte-encoding": "^1.0.3",
     "electron-context-menu": "^3.3.0",
     "electron-log": "^4.4.8",
-    "electron-updater": "^5.3.0",
+    "electron-updater": "^6.1.1",
     "fs-jetpack": "^4.3.1",
     "glob": "^7.2.0",
     "google": "^2.1.0",
@@ -162,8 +162,8 @@
     "chai": "^4.3.6",
     "copy-webpack-plugin": "^11.0.0",
     "css-loader": "^6.7.1",
-    "electron": "24.3.1",
-    "electron-builder": "23.6.0",
+    "electron": "26.1.0",
+    "electron-builder": "24.6.3",
     "eslint": "^8.17.0",
     "eslint-plugin-react": "^7.30.0",
     "express": "^4.18.1",

src/app/app.js

@@ -15,7 +15,8 @@ window.__ = __
 if(typeof WEB !== 'undefined')
 {
 	const io = require("socket.io-client");
-	window.torrentSocket = io(document.location.protocol + '//' + document.location.hostname + (process.env.NODE_ENV === 'production' ? '/' : ':8095/'));
+	// window.torrentSocket = io(document.location.protocol + '//' + document.location.hostname + (process.env.NODE_ENV === 'production' ? '/' : ':8095/'));
+  window.torrentSocket = io(document.location.protocol + '//' + document.location.host + '/');
 	const emit = window.torrentSocket.emit.bind(window.torrentSocket);
 	window.torrentSocket.emit = (...data) => {
 		let id;
@@ -85,7 +86,7 @@ else
 
 	ipcRenderer.on('url', (event, url) => {
 		console.log('url', url)
-		router(url)    
+		router(url)
 	});
 
 }

src/app/config-page.js

@@ -216,6 +216,16 @@ export default class ConfigPage extends Page {
 								}}
 							/>
 
+							<Toggle
+								style={{marginTop: '10px'}}
+								label={__('REST API support')}
+								toggled={this.options.restApi}
+								onToggle={(e, checked) => {
+									this.options.restApi = checked
+									this.forceUpdate()
+								}}
+							/>
+
 						</div>
 					</Tab>
 					<Tab value='p2p' label={__("P2P settings")} icon={<SvgIcon viewBox="0 0 47 47">

src/background/api.js

@@ -56,7 +56,7 @@ module.exports = async ({
 					downloaded: download.downloaded,
 					progress: download.progress,
 					downloadSpeed: download.downloadSpeed,
-                    
+
 					removeOnDone: download.removeOnDone,
 					paused: torrent.paused || torrent._paused
 				}
@@ -79,10 +79,10 @@ module.exports = async ({
 		return torrents
 	}
 
-	const mergeTorrentsWithDownloadsFn = (Fn, copy) => (...args) => { 
+	const mergeTorrentsWithDownloadsFn = (Fn, copy) => (...args) => {
 		const callback = args[args.length - 2]
 		const rest = args.slice(0, -2)
-		Fn(...rest, (data) => callback(mergeTorrentsWithDownloads(data, copy)), args[args.length - 1]) 
+		Fn(...rest, (data) => callback(mergeTorrentsWithDownloads(data, copy)), args[args.length - 1])
 	}
 
 	const downloadFilesList = (torrent) => torrent.files.map((file, index) => ({
@@ -215,7 +215,7 @@ module.exports = async ({
 		p2p.on('randomTorrents', (nil, callback) => {
 			if(typeof callback != 'function')
 				return;
-    
+
 			const cpu = cpuUsage()
 			const limit = Math.max(1, 5 - (cpu / 20) | 0)
 
@@ -224,19 +224,19 @@ module.exports = async ({
 					callback(undefined)
 					return;
 				}
-    
+
 				let hashes = {}
 				for(const torrent of torrents)
 				{
 					delete torrent.id
 					hashes[torrent.hash] = torrent
 				}
-    
+
 				const inSql = Object.keys(hashes).map(hash => sphinx.escape(hash)).join(',');
 				sphinxSingle.query(`SELECT * FROM files WHERE hash IN(${inSql})`, (error, files) => {
 					for(const file of files)
 						hashes[file.hash].filesList = parseTorrentFiles(file);
-                    
+
 					callback(Object.values(hashes))
 				})
 			})
@@ -285,7 +285,7 @@ module.exports = async ({
 		if(typeof callback != 'function')
 			return;
 
-		if(!text || text.length <= 2) {
+		if(!text || text.length < 2) {
 			callback(undefined);
 			return;
 		}
@@ -347,7 +347,7 @@ module.exports = async ({
 			{
 				logT('search', 'get torrent via infohash with dht')
 				// 3 try to get torrent from metadata
-				const getTorrentMetadata = (tryCount = 4) => {
+				const getTorrentMetadata = (tryCount = 8) => {
 					if(tryCount <= 0) {
 						logT('search', 'dht NOT found anything with dht', text);
 						return
@@ -356,7 +356,7 @@ module.exports = async ({
 					dhtCheckTimeout = setTimeout(() => {
 						lock = true
 						getTorrentMetadata(--tryCount)
-					}, 8000);
+					}, 16000);
 					torrentClient.getMetadata(text, (torrent) => {
 						if(lock) {
 							logT('search', 'this dht response not actual for', text);
@@ -407,7 +407,7 @@ module.exports = async ({
 		if(typeof callback != 'function')
 			return;
 
-		if(!text || text.length <= 2) {
+		if(!text || text.length < 2) {
 			callback(undefined);
 			return;
 		}
@@ -455,7 +455,7 @@ module.exports = async ({
 				for(const torrent of torrents)
 				{
 					search[torrent.hash] = Object.assign(baseRowData(torrent), search[torrent.hash])
-                    
+
 					// temporary ignore adult content in search (workaroud)
 					if(safeSearch && search[torrent.hash].contentCategory == 'xxx')
 						delete search[torrent.hash]
@@ -531,7 +531,7 @@ module.exports = async ({
 				where += ' and `added` > ' + (Math.floor(Date.now() / 1000) - (60 * 60 * 24 * 30))
 			}
 		}
-        
+
 		const query = `SELECT * FROM torrents WHERE seeders > 0 and contentCategory != ${torrentCategoryId('xxx')} ${where} ORDER BY seeders DESC LIMIT ${index},${limit}`;
 		if(topCache[query])
 		{
@@ -543,7 +543,7 @@ module.exports = async ({
 				callback(undefined)
 				return;
 			}
-            
+
 			rows = rows.map((row) => baseRowData(row));
 			topCache[query] = rows;
 			callback(rows);
@@ -641,7 +641,7 @@ module.exports = async ({
 		delete copyConfig['load'];
 		delete copyConfig['reload'];
 		send('configChanged', copyConfig)
-        
+
 		if(typeof callback === 'function')
 			callback(true)
 	});
@@ -697,7 +697,7 @@ module.exports = async ({
 			send('filesReady', torrent.infoHash, downloadFilesList(torrent))
 		})
 
-		torrent.on('done', () => { 
+		torrent.on('done', () => {
 			logT('downloader', 'download done', torrent.infoHash)
 			progress(0) // update progress
 			// remove torrent if marked
@@ -709,7 +709,7 @@ module.exports = async ({
 						logT('downloader', 'download removing error', err)
 						return
 					}
-        
+
 					delete torrentClientHashMap[torrent.infoHash]
 					send('downloadDone', torrent.infoHash)
 				})
@@ -793,10 +793,10 @@ module.exports = async ({
 			}
 			torrent.updateFilesSelection()
 		}
-        
+
 		torrent.updateFilesSelection = () => {
 			torrent.deselect(0, torrent.pieces.length - 1, false)
-        
+
 			for(const file of torrent.files)
 			{
 				const {selected} = file
@@ -825,7 +825,7 @@ module.exports = async ({
 			logT('downloader', 'cant find torrent for removing', hash)
 			return
 		}
-        
+
 		const torrent = torrentClient.get(id)
 		if(!torrent) {
 			logT('downloader', 'no torrent for update founded')
@@ -913,7 +913,7 @@ module.exports = async ({
 			downloaded: torrent.downloaded,
 			progress: torrent.progress,
 			downloadSpeed: torrent.downloadSpeed,
-            
+
 			removeOnDone: torrent.removeOnDone,
 			paused: torrent.paused || torrent._paused
 		})))
@@ -1018,11 +1018,11 @@ module.exports = async ({
 
 	// store torrent to feed
 	await feed.load()
-	Object.defineProperty(p2p.info, 'feed', { 
+	Object.defineProperty(p2p.info, 'feed', {
 		enumerable: true,
 		get: () => feed.size()
 	});
-	Object.defineProperty(p2p.info, 'feedDate', { 
+	Object.defineProperty(p2p.info, 'feedDate', {
 		enumerable: true,
 		get: () => feed.feedDate
 	});
@@ -1032,9 +1032,9 @@ module.exports = async ({
 
 		if(!temp || !temp.torrent)
 			return
-    
+
 		const { torrent } = temp
-        
+
 		if(torrent.hash !== record.torrentHash)
 			return
 
@@ -1053,7 +1053,7 @@ module.exports = async ({
 		// update feed only on some good info
 		if(torrent.good < 1)
 			return
-        
+
 		feed.add(torrent)
 		send('feedUpdate', {
 			feed: feed.feed
@@ -1083,7 +1083,7 @@ module.exports = async ({
 				peer.emit('feed', null, (remoteFeed) => {
 					if(!remoteFeed)
 						return
-            
+
 					if(Array.isArray(remoteFeed) || !remoteFeed.feed)
 						return // old version call
 
@@ -1091,7 +1091,7 @@ module.exports = async ({
 						logT('feed', 'remote feed have more torrent that needed: ', remoteFeed.feed.length, ' > ', feed.max);
 						remoteFeed.feed = remoteFeed.feed.slice(0, feed.max);
 					}
-            
+
 					if(remoteFeed.feed.length > feed.size() || (remoteFeed.feed.length == feed.size() && remoteFeed.feedDate > feed.feedDate))
 					{
 						logT('feed', 'replace our feed with remote feed')
@@ -1107,5 +1107,5 @@ module.exports = async ({
 			}
 		}
 	})
-    
-}
+
+}

src/background/background.js

@@ -102,6 +102,11 @@ if(portative)
 process.on('unhandledRejection', r => logTE('system', 'Rejection:', r));
 process.on('uncaughtException', (err, origin) => logTE('system', 'Exception:', err, 'Origin:', origin));
 
+if (env.name !== "production" && (!fs.existsSync(__dirname + '/../imports') || fs.readdirSync(__dirname + '/../imports').length == 0)) {
+	logTE('system', 'You are not clonned submodules correctly, please use git clone --recurse-submodules https://github.com/DEgITx/rats-search.git');
+	process.exit(1);
+}
+
 const gotTheLock = app.requestSingleInstanceLock()
 if (!gotTheLock) {
 	logT('app', 'closed because of second application')

src/background/server.js

@@ -40,6 +40,11 @@ if(majorVersion < 8)
 	process.exit(1);
 }
 
+if (!fs.existsSync(__dirname + '/../../imports') || fs.readdirSync(__dirname + '/../../imports').length == 0) {
+	logTE('system', 'You are not clonned submodules correctly, please use git clone --recurse-submodules https://github.com/DEgITx/rats-search.git');
+	process.exit(1);
+}
+
 app.use(express.static('web'));
 
 appConfig.restApi = true;
@@ -58,11 +63,32 @@ io.on('connection', (socket) =>
 	}
 })
 
+let responceRestQueue = [];
+
+if (appConfig.restApi) {
+	app.get('/api/queue', (req, res) => {
+		const uniqueId = Math.random().toString(16).slice(2) + '_' + (new Date()).getTime();
+		logT('rest', 'queue responce', uniqueId, 'size', responceRestQueue.length);
+		res.send({id: uniqueId, queue: responceRestQueue})
+		// clear queue after the read of json queue
+		responceRestQueue = [];
+	});
+}
+
 const start = async () => 
 {
 	({ sphinx } = await startSphinx(() => {
 		dbPatcher(() => {
-			spider = new spiderCall((...data) => io.sockets.emit(...data), (message, callback) => {
+			spider = new spiderCall((...data) => { 
+				if (appConfig.restApi) {
+					if (responceRestQueue.length < 1000) {
+						responceRestQueue.push(data);
+					} else {
+						logTE('rest', 'max 1000 queue records, please use /api/queue to clean records')
+					}
+				}
+				return io.sockets.emit(...data) 
+			}, (message, callback) => {
 				socketMessages[message] = callback
 				if (appConfig.restApi) {
 					app.get('/api/' + message, (req, res) => {

translations/cn.json

@@ -198,6 +198,7 @@
         "File": "文件",
         "Folder": "文件夾",
         "Generate": "產生",
-        "Dark mode theme": "Dark mode theme"
+        "Dark mode theme": "Dark mode theme",
+        "REST API support": "REST API support"
     }
 }

translations/en.json

@@ -198,6 +198,7 @@
         "File": "File",
         "Folder": "Folder",
         "Generate": "Generate",
-        "Dark mode theme": "Dark mode theme"
+        "Dark mode theme": "Dark mode theme",
+        "REST API support": "REST API support"
     }
 }

translations/ru.json

@@ -198,6 +198,7 @@
         "File": "Файл",
         "Folder": "Папка",
         "Generate": "Сгенерировать",
-        "Dark mode theme": "Темная тема"
+        "Dark mode theme": "Темная тема",
+        "REST API support": "REST API support"
     }
 }

translations/ua.json

@@ -198,6 +198,7 @@
         "File": "Файл",
         "Folder": "Папка",
         "Generate": "Згенерувати",
-        "Dark mode theme": "Dark mode theme"
+        "Dark mode theme": "Dark mode theme",
+        "REST API support": "REST API support"
     }
 }