12eefa7557
crawler: Handle no new users case, reduce poll rate.
2018-01-08 08:07:31 +10:00
1a0c91e13a
server: Retrieve new users from database.
...
No need to go out and fetch them ourselves anymore.
2018-01-08 08:05:15 +10:00
2b2a01852d
crawler: Periodically refresh up to 50 users.
2018-01-08 07:47:34 +10:00
9d19a3d44b
crawler: Set an event when new users are added.
2018-01-08 07:32:51 +10:00
4ef3d9a9d9
crawler: Keep reading older users in the background.
2018-01-08 07:29:43 +10:00
8f5bdf9299
crawler: Drop old user retrieval.
...
Realised that this won't work the way I had planned, because my own UID
is far lower than the block I've pulled in: it'll see that and start
there leaving a huge gap.
Plus, it just returned `users: 0`, not useful.
2018-01-07 22:30:44 +10:00
9687e3f637
crawler: Reverse start and end for old user retrieval.
2018-01-07 22:28:16 +10:00
4bf473713e
hadapi: Handle socket.gaierror(EAGAIN)
2018-01-07 22:26:34 +10:00
53686bf5d7
crawler: Quietly retrieve more users in the background.
2018-01-07 22:23:48 +10:00
f212ea3b04
crawler: Validate user URIs
...
Seems the users still show up in the API but are since "deleted". So do
a check to see if the link's valid.
2018-01-07 21:27:40 +10:00
daee438af7
hadapi: Fix retrieve users by range
2018-01-07 20:51:08 +10:00
e8d67d872c
hadapi: Use range query to retrieve users.
2018-01-07 20:48:10 +10:00
6ec6116cd9
main.js: Delay auto-marking "legit" users.
...
Just in case the user spots something at the last minute.
2018-01-07 19:12:46 +10:00
032650601b
crawler: Also ignore Google+
2018-01-07 18:47:57 +10:00
183dc3eef5
crawler: Fix inverted logic
2018-01-07 18:43:48 +10:00
9ae48a4f7b
main.js: Automatically mark 'auto_legit'
...
The number of false negatives here has been tiny, so this will make life
a little more convenient.
2018-01-07 18:32:59 +10:00
e484385a8e
crawler: Ignore 'github' or 'twitter' links.
...
By far the most common on this site. Not anomalous.
2018-01-07 18:29:16 +10:00
83e7785e19
main.js: Add legit/suspect buttons.
...
Note, they only work if you're an admin, otherwise they do nothing.
2018-01-07 17:33:08 +10:00
8a9a1608f5
server: Implement classification endpoint.
2018-01-07 17:30:51 +10:00
6eb99f2653
hadapi, util: Re-locate body parsing
...
We'll need it in the server for requesting POST/PUT bodies too.
2018-01-07 16:55:32 +10:00
6b5c4e8b6a
crawler: Periodically refresh admin group members.
2018-01-07 16:44:37 +10:00
9ad7b7dd86
crawler: Refresh admin group at start-up.
2018-01-07 16:34:31 +10:00
dcfe89fba5
crawler: Store project ID
2018-01-07 16:17:30 +10:00
db39ddbbcb
server: Pass through project ID from command line.
2018-01-07 16:17:15 +10:00
f16437cf38
crawler: Skip users that have been classified by a human
2018-01-07 16:05:07 +10:00
2083e66894
main.js: Show groups and tags of users.
2018-01-07 15:43:35 +10:00
906aaa97be
server: Expose user groups and tags
2018-01-07 15:34:08 +10:00
dcc61c644d
crawler: Automatically file users into groups.
2018-01-07 15:25:39 +10:00
d3a9599822
crawler: Drop redundant page_last_refresh
2018-01-07 15:20:05 +10:00
42432241d3
model: Fix reference on Tags
2018-01-07 15:16:50 +10:00
3de3d25b8d
server: Drop unused classes
2018-01-07 14:58:12 +10:00
03ef9f8a8a
crawler: Drop unused classes
2018-01-07 14:57:33 +10:00
2deb05dccc
db.db: Drop unused classes
2018-01-07 14:57:06 +10:00
2dbf81f6eb
db.model: Model many-to-many properly.
2018-01-07 14:56:56 +10:00
a462f39358
server: Convert date/time to ISO format
2018-01-07 14:39:47 +10:00
c7a6529f17
main.js: Show user creation date
2018-01-07 14:38:35 +10:00
1369a6089f
server: Show user creation date.
2018-01-07 14:38:23 +10:00
0bfd1ec6cc
crawler: Update creation date on existing users.
2018-01-07 14:35:19 +10:00
2accdee2a3
crawler: Tweak newest user page refresh
...
- Use the database to persist when we last checked a page, so we don't
flog HAD's site unnecessarily.
- Bump the starting offset to 1 and the timeout to a day.
2018-01-07 14:22:17 +10:00
4c979ecc43
db.model: Add page refresh metadata.
2018-01-07 14:10:04 +10:00
b3c5c5e304
crawler: Add user creation date
2018-01-07 14:06:25 +10:00
db3c47dd89
db.model: Add creation date to user
2018-01-07 14:03:25 +10:00
4861579deb
crawler: Skip pages loaded in the last hour.
...
Beyond page 10, if we've loaded that page in the last hour, assume
nothing has changed.
2018-01-07 12:28:09 +10:00
650178fcf3
main.js: Use page number from response.
2018-01-07 11:39:50 +10:00
786cc1339a
crawler: Fix retrieval when existing users seen.
2018-01-07 11:39:31 +10:00
6be5e9a297
server: Retrieve page from fetch_new_users.
2018-01-07 11:36:00 +10:00
b09494b726
crawler: Report current page, move to next if no users.
2018-01-07 11:35:29 +10:00
83aa52a117
crawler: Don't inspect existing users.
...
When browsing through the "new user" list, skip accounts that have been
inspected already.
2018-01-07 11:30:48 +10:00
15c9970ccc
main.js: Show profile information if present.
2018-01-07 10:15:27 +10:00
96f2e1c63f
server: Fix reference to crawler in avatar retrieval.
2018-01-07 10:08:20 +10:00
