cd1e49c28e
hadapi: Tweak profile page resolution
2019-01-01 18:41:50 +10:00
27d5c65192
hadapi: Re-work user ID scan.
...
This is a work-around to the fact that the API throws error 500 when you
try to retrieve the "newest" users (in spite of the fact that those are
generally the users that are of interest from a spam-prevention point of
view).
This is a last attempt to try and make this work. Next time I just shut
down the project and let Hackaday.io drown in its spam, I spend too many
hours a week trying to clean up this crap and my own projects are
suffering because of it.
2019-01-01 18:00:13 +10:00
3f47bfbea8
hadapi: Use extdlog for logging.
2018-08-04 12:22:16 +10:00
0f3fb6b6ad
hadapi: Re-work rate limit handling
...
To manage the rate limit, easiest way is to permit one request at a
time, so that we don't pummel the server with requests and get it upset.
For this, we don't need such an extravagant set-up.
2018-07-01 18:36:16 +10:00
7ef0e8c870
hadapi: Increase back-off when connection is reset.
2018-07-01 14:55:35 +10:00
d750dabba3
hadapi: Handle no response
2018-07-01 14:51:44 +10:00
ebb0394c60
hadapi: Add function to retrieve user IDs alone.
...
Without returning the user info, so we can quickly skip the users we
already have.
2018-07-01 12:49:42 +10:00
35e36af073
hadapi: Don't show content of all responses.
...
Some are JPEGs and PNGs, these are not meaningful to display in logs.
2018-06-03 11:13:15 +10:00
88bd611e8d
hadapi: Handle removed per_page parameter
...
It's meaningless for the workaround code.
2018-06-02 21:01:28 +10:00
f6b6932ca8
hadapi: Actually return the response
2018-06-02 20:21:45 +10:00
e3ca7b7a19
hadapi: Expose retrieval function that respects limits.
...
This basically is a wrapper around the HTTP client, and tries to ensure
that *all* requests meet the request limits to avoid getting blocked.
2018-06-02 16:49:53 +10:00
841d89898e
hadapi: Use workaround code for users sorted by newest.
...
This has been broken for a while now, and doesn't look like it'll ever
get fixed, so just work around it here.
2018-06-02 16:38:07 +10:00
1710dfdbc0
hadapi: Treat connection reset as a sign to back-off.
...
Also reduce our query rate a little bit to avoid the problem.
2018-06-02 16:36:01 +10:00
ae40f23d2d
hadapi: Report error responses
2018-06-02 11:09:53 +10:00
4cdd8fcc25
hadapi: Drop range query in work-around
2018-05-02 21:26:07 +10:00
bc57dc1825
hadapi: Add debugging
2018-05-02 20:50:07 +10:00
b00f860a00
hadapi: Have some more patience
2018-03-02 20:47:44 +10:00
eee087843a
hadapi: Fix typo in _project_query_opts.
2018-03-01 21:11:06 +10:00
5cd0afb196
hadapi: Re-work forbidden handling.
...
Rather than a dumb wait of an hour, we simply set a flag that's exposed
elsewhere. Our background tasks can check that flag instead and inhibit
their operations themselves.
If a user action that requires API access, succeeds, the flag is
cleared and the background tasks resume.
The background tasks auto-retry after an hour.
2018-01-11 20:40:04 +10:00
72feab73e3
hadapi: Wait before acquiring semaphore
2018-01-10 08:28:00 +10:00
034e9026c8
hadapi: Don't wait after forbidden for authentication calls.
...
The user is *not* going to wait an hour.
2018-01-10 08:24:50 +10:00
f01fa59c98
hadapi: If we receive 403 Forbidden, back off for an hour.
2018-01-09 07:16:21 +10:00
4bf473713e
hadapi: Handle socket.gaierror(EAGAIN)
2018-01-07 22:26:34 +10:00
daee438af7
hadapi: Fix retrieve users by range
2018-01-07 20:51:08 +10:00
e8d67d872c
hadapi: Use range query to retrieve users.
2018-01-07 20:48:10 +10:00
6eb99f2653
hadapi, util: Re-locate body parsing
...
We'll need it in the server for requesting POST/PUT bodies too.
2018-01-07 16:55:32 +10:00
ecaf40a65d
hadapi: Use a set, not a list for batch reads.
...
So we don't read duplicates.
2018-01-07 00:22:01 +10:00
c9ea92d165
hadapi: Don't specify sortby=newest
...
Even when retrieving batch IDs, it cocks up.
2018-01-06 23:12:31 +10:00
e5b2531bb0
hadsh: Fix generation of user ID list
2018-01-06 23:08:15 +10:00
c5374f9753
hadsh: Make get_users a coroutine.
2018-01-06 23:07:00 +10:00
cab4b24b04
hadapi: Fix missed perpage typo
2018-01-06 23:03:27 +10:00
79a5782552
hadapi: Hack around bug in hackaday.io API
...
So it appears that `sortby=newest` is broken for the user list. For the
short term, I can work-around this by screen-scraping
https://hackaday.io/hackers?sort=newest and doing a batch read of the
IDs.
2018-01-06 23:02:07 +10:00
2e9907f81b
crawler, hadapi: Fix typo in per_page.
2018-01-06 21:54:59 +10:00
49bd53c5e9
hadapi: Fix typo in _user_query_opts
2018-01-06 21:13:35 +10:00
d4837338cc
hadapi: Don't include second query string
2018-01-06 18:45:29 +10:00
05cef594a8
hadapi: Show query arguments before URL encoding
2018-01-06 18:43:45 +10:00
220182733e
hadapi: Trace API calls
2018-01-06 18:42:08 +10:00
465a955dcf
hadapi: Fix quote_plus reference
2018-01-06 18:34:24 +10:00
4444ae958e
hadapi: Fix copy-paste error
2018-01-06 18:26:13 +10:00
129fb8acf0
hadsh: Add missed bracket
2018-01-06 17:56:51 +10:00
4ae4aa701d
hadapi: Expose authentication URI
2018-01-06 17:32:16 +10:00
5265def117
hadapi: Add rate limit handling.
2018-01-06 17:14:49 +10:00
a07a2755be
hadapi: Add beginnings of API wrapper
...
Handles users and projects for now.
2018-01-06 16:07:45 +10:00
