tscrape, branch HEAD

README: small improvement

2021-07-20T13:20:17Z

commit 291870afea3ad56366e82efaceef0f1288b34018 parent f0b2764a08ed2e73dc098f7b4792b81284624944 Author: Hiltjo Posthuma Date: Tue, 20 Jul 2021 15:20:17 +0200 README: small improvement

README: fix twitter to Atom (sfeed) example

2021-07-20T12:49:43Z

commit f0b2764a08ed2e73dc098f7b4792b81284624944 parent ca46299e72f8e28b0f9211b55ccc65e62eaa9306 Author: Hiltjo Posthuma Date: Tue, 20 Jul 2021 14:49:43 +0200 README: fix twitter to Atom (sfeed) example Thanks Nathaniel for reporting it!

bump LICENSE year

2021-03-20T14:53:18Z

commit ca46299e72f8e28b0f9211b55ccc65e62eaa9306 parent 34ed10b505310477e31b608a28470e2e997eb4ea Author: Hiltjo Posthuma Date: Sat, 20 Mar 2021 15:53:18 +0100 bump LICENSE year

update static bearer token

2021-03-19T10:25:29Z

commit 34ed10b505310477e31b608a28470e2e997eb4ea parent 76a1794bec9f2e64572537abce0258827ce93a80 Author: Hiltjo Posthuma Date: Fri, 19 Mar 2021 11:25:29 +0100 update static bearer token This uses the same bearer as youtube-dl. Thanks frobnitz!

bump version to 0.6

2020-10-01T18:23:15Z

commit 76a1794bec9f2e64572537abce0258827ce93a80 parent cdfc383b166a112646ab21596442d8fb976bc311 Author: Hiltjo Posthuma Date: Thu, 1 Oct 2020 20:23:15 +0200 bump version to 0.6

tscrape: remove debugging comments

2020-10-01T18:22:18Z

commit cdfc383b166a112646ab21596442d8fb976bc311 parent acef6f3969b21a48512c389febf2e6ca9089ebfc Author: Hiltjo Posthuma Date: Thu, 1 Oct 2020 20:22:18 +0200 tscrape: remove debugging comments

tscraperc.example: change feeds

2020-10-01T18:21:44Z

commit acef6f3969b21a48512c389febf2e6ca9089ebfc parent dc41402d290af10b6695824342339cba108cc6c0 Author: Hiltjo Posthuma Date: Thu, 1 Oct 2020 20:21:44 +0200 tscraperc.example: change feeds

tscrape_update: escape { and } in sed expression

2020-10-01T18:10:06Z

commit dc41402d290af10b6695824342339cba108cc6c0 parent 0ec2e56f6971b7e33e84fc35aeaa796f9044554e Author: Hiltjo Posthuma Date: Thu, 1 Oct 2020 20:10:06 +0200 tscrape_update: escape { and } in sed expression Thanks anthk for reporting it.

tscrape_update: increase timeline limit from 50 to 100

2020-06-19T12:05:06Z

commit 0ec2e56f6971b7e33e84fc35aeaa796f9044554e parent ff8d2ecaed4cb56e6cc1ccdc4a43e1a3e45eb61f Author: Hiltjo Posthuma Date: Fri, 19 Jun 2020 14:05:06 +0200 tscrape_update: increase timeline limit from 50 to 100

refactor urls into general replacement function and replace some HTML entities

2020-06-06T22:08:42Z

commit ff8d2ecaed4cb56e6cc1ccdc4a43e1a3e45eb61f parent 62a3853f6428208e5be727175479ebcede127497 Author: Hiltjo Posthuma Date: Sun, 7 Jun 2020 00:08:42 +0200 refactor urls into general replacement function and replace some HTML entities

README: some rewording

2020-06-06T10:51:50Z

commit 62a3853f6428208e5be727175479ebcede127497 parent 5ca572b963ed7b64485d016eec950ca3a646d107 Author: Hiltjo Posthuma Date: Sat, 6 Jun 2020 12:51:50 +0200 README: some rewording

rm code thats not needed anymore and a debug/test comment

2020-06-06T10:47:51Z

commit 5ca572b963ed7b64485d016eec950ca3a646d107 parent 86910f72369e655ce2db017e64646def543627b9 Author: Hiltjo Posthuma Date: Sat, 6 Jun 2020 12:47:51 +0200 rm code thats not needed anymore and a debug/test comment

Expand all photos URLs

2020-06-06T00:23:37Z

commit 86910f72369e655ce2db017e64646def543627b9 parent 9cc1b7e985affd8764385d3d6cf6476804230cdd Author: Leonardo Taccari Date: Sat, 6 Jun 2020 02:23:37 +0200 Expand all photos URLs

Expand extended_entities URLs in retweets

2020-06-05T23:50:22Z

commit 9cc1b7e985affd8764385d3d6cf6476804230cdd parent d270b9dc10bc3d05b096f2dd34256dc9b962b951 Author: Leonardo Taccari Date: Sat, 6 Jun 2020 01:50:22 +0200 Expand extended_entities URLs in retweets

Use extended_entities instead of entities

2020-06-05T23:40:36Z

commit d270b9dc10bc3d05b096f2dd34256dc9b962b951 parent e2c4c24378d937edd6f9d717267d9f08b268df78 Author: Leonardo Taccari Date: Sat, 6 Jun 2020 01:40:36 +0200 Use extended_entities instead of entities Possible medias could be omitted in entities but present in extended_entities.

Further expand URLs in retweets

2020-06-05T23:11:57Z

commit e2c4c24378d937edd6f9d717267d9f08b268df78 parent d204e2373cc9f7e3f3afa3d4f2afb7976f67b4ae Author: Leonardo Taccari Date: Sat, 6 Jun 2020 01:11:57 +0200 Further expand URLs in retweets

update man pages a bit for the recent changes

2020-06-05T15:19:39Z

commit d204e2373cc9f7e3f3afa3d4f2afb7976f67b4ae parent 7ed31cfe093fa52711f0bdd407abc1e4d67f11b3 Author: Hiltjo Posthuma Date: Fri, 5 Jun 2020 17:19:39 +0200 update man pages a bit for the recent changes

replace all whitespace by a single space

2020-06-05T15:18:40Z

commit 7ed31cfe093fa52711f0bdd407abc1e4d67f11b3 parent d4929d30226753c84a2ab5d8b703dd7df69eb4c1 Author: Hiltjo Posthuma Date: Fri, 5 Jun 2020 17:18:40 +0200 replace all whitespace by a single space

Fix an off-by-one in printexpand() by leot

2020-06-05T15:17:17Z

commit d4929d30226753c84a2ab5d8b703dd7df69eb4c1 parent 57259fbc5868fc4c8225640aaca8fd496dd6fc6c Author: Hiltjo Posthuma Date: Fri, 5 Jun 2020 17:17:17 +0200 Fix an off-by-one in printexpand() by leot Thanks leot!

simplify parsejson error checking

2020-06-05T15:16:14Z

commit 57259fbc5868fc4c8225640aaca8fd496dd6fc6c parent e386c4bbe4e59ce447c921b9063318e98318c0f1 Author: Hiltjo Posthuma Date: Fri, 5 Jun 2020 17:16:14 +0200 simplify parsejson error checking

replace newline to space instead of removing them

2020-06-05T15:15:34Z

commit e386c4bbe4e59ce447c921b9063318e98318c0f1 parent c3e76b0f57c58b284cd13ce008c082525c8ee28a Author: Hiltjo Posthuma Date: Fri, 5 Jun 2020 17:15:34 +0200 replace newline to space instead of removing them

work-in-progress: support the new Twitter site

2020-06-05T12:51:58Z

commit c3e76b0f57c58b284cd13ce008c082525c8ee28a parent 663dab7d9883a291ed570a743fb89a16e1a01d85 Author: Hiltjo Posthuma Date: Fri, 5 Jun 2020 14:51:58 +0200 work-in-progress: support the new Twitter site Scraping doesn't work anymore. Use the Twitter JSON API. Major thanks to leot for helping with this.

fix typo

2020-06-01T10:17:51Z

commit 663dab7d9883a291ed570a743fb89a16e1a01d85 parent 4eb084c604ac84247e704bd05a4737c8679461e2 Author: Hiltjo Posthuma Date: Mon, 1 Jun 2020 12:17:51 +0200 fix typo

bump version to 0.5

2020-05-04T11:37:42Z

commit 4eb084c604ac84247e704bd05a4737c8679461e2 parent b83b7243ecfafc432cf1807f69c9ab30068fbd1f Author: Hiltjo Posthuma Date: Mon, 4 May 2020 13:37:42 +0200 bump version to 0.5

style.css: improve horizontal scrolling for long titles/small windows

2020-05-02T13:15:50Z

commit b83b7243ecfafc432cf1807f69c9ab30068fbd1f parent 8e11f80c105f02b55641d9068d03448a6ffc1ea3 Author: Hiltjo Posthuma Date: Sat, 2 May 2020 15:15:50 +0200 style.css: improve horizontal scrolling for long titles/small windows

util: printescape(): it's enough to set a flag, no counter needed

2020-03-20T18:09:49Z

commit 8e11f80c105f02b55641d9068d03448a6ffc1ea3 parent 9a0c90f26b3eef023910a699c7385b709a01ce90 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 19:09:49 +0100 util: printescape(): it's enough to set a flag, no counter needed

documentation: no need to explicitly set HTTP/1

2020-03-20T17:37:11Z

commit 9a0c90f26b3eef023910a699c7385b709a01ce90 parent e03dec6b97f74451786b3698c9fab1d0d45b9cd9 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 18:37:11 +0100 documentation: no need to explicitly set HTTP/1 (HTTP/2+ still sucks though)

tscrape_update: tiny style fix in comment

2020-03-20T17:18:17Z

commit e03dec6b97f74451786b3698c9fab1d0d45b9cd9 parent c427d54a874baa5c3f0cd0d725b90efa27cf34d2 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 18:18:17 +0100 tscrape_update: tiny style fix in comment

tscrape_html: use a

 section per feed

2020-03-20T17:15:12Z

commit c427d54a874baa5c3f0cd0d725b90efa27cf34d2 parent b0413f42bd2bc31cbbb5e338093de51b94cfd028 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 18:15:12 +0100 tscrape_html: use a

 section per feed

This improves output with Dillo, w3m and possibly other simple browsers.

- Dillo has a bug where it resets its block-style after  when it is inside
  . Then it ignores newlines (inside ) and the links are inlined.
- w3m does not have a (line)margin for h2.

man page improvements (sync)

2020-03-20T11:03:26Z

commit b0413f42bd2bc31cbbb5e338093de51b94cfd028 parent 423d3f5ad6023be3eb50ebe2f9504309bfe3d940 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 12:03:26 +0100 man page improvements (sync) - tscraperc.5: use the same order as executed in the tscrape_update file. - tscraperc.5: reference curl, which are optional, but used by default. - tscraperc.5: use a .Sh VARIABLES section for tscrapepath and maxjobs. - tscrape_update.1: split config format-specific documentation and reference it. - just use the term "url" instead of "uri". - shorten some texts, increasing readability. - document exit status of tools. fix: - do not reference RSS/Atom.

tscrape_html/tscrape_plain: like sfeed, skip when a timestamp is empty/invalid

2020-03-20T11:02:41Z

commit 423d3f5ad6023be3eb50ebe2f9504309bfe3d940 parent d34cbfb8d0bc3d985ce58ebfa3034115a0ef323d Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 12:02:41 +0100 tscrape_html/tscrape_plain: like sfeed, skip when a timestamp is empty/invalid

README: don't set LC_ALL for awk example

2020-03-20T11:01:35Z

commit d34cbfb8d0bc3d985ce58ebfa3034115a0ef323d parent 573905aec2e99fbe31a1cabe5864853ef9015a41 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 12:01:35 +0100 README: don't set LC_ALL for awk example

sync printutf8pad from sfeed

2020-03-20T11:00:16Z

commit 573905aec2e99fbe31a1cabe5864853ef9015a41 parent 426522824e719e081c9c5e47ba8771779b0fdc85 Author: Hiltjo Posthuma Date: Fri, 20 Mar 2020 12:00:16 +0100 sync printutf8pad from sfeed changes: - util: printutf8pad: proper counting of multiwidth characters for example the string "\xef\xbc\xb5". - optimization

ignore possible consecutive `js-stream-item' when parsing a single tweet

2020-03-19T22:52:19Z

commit 426522824e719e081c9c5e47ba8771779b0fdc85 parent 6654f1b01d68e2b2ff7aa660cd678c1cba4d062f Author: Leonardo Taccari Date: Thu, 19 Mar 2020 23:52:19 +0100 ignore possible consecutive `js-stream-item' when parsing a single tweet

bump version to 0.4

2020-02-23T19:36:28Z

commit 6654f1b01d68e2b2ff7aa660cd678c1cba4d062f parent e22be126773aae620aadb29b9757824dc1060868 Author: Hiltjo Posthuma Date: Sun, 23 Feb 2020 20:36:28 +0100 bump version to 0.4

tscrape_update: don't preserve permissions of tmp files by moving, so copy

2020-02-23T19:18:01Z

commit e22be126773aae620aadb29b9757824dc1060868 parent 1b03719564afdb0ba61fa1599d0cb796e10d6ed9 Author: Hiltjo Posthuma Date: Sun, 23 Feb 2020 20:18:01 +0100 tscrape_update: don't preserve permissions of tmp files by moving, so copy noticed on DragonFlyBSD where it prints a warning when moving the file from /tmp. To reproduce it: touch /tmp/file mv /tmp/file ~/ On other systems this would not print a warning, but it would preserve the group permissions etc.

cleanup some more includes

2020-02-01T14:56:04Z

commit 1b03719564afdb0ba61fa1599d0cb796e10d6ed9 parent 07ed8752bbdf330e170b53dc29f35044cbbbe958 Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:56:04 +0100 cleanup some more includes

rm example.sh

2020-02-01T14:30:37Z

commit 07ed8752bbdf330e170b53dc29f35044cbbbe958 parent 64d62523dd2ee855258cef621a0942875910e416 Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:30:37 +0100 rm example.sh the field order was changed and tscrape_plain is a better use now.

bump LICENSE to 2020

2020-02-01T14:13:52Z

commit 64d62523dd2ee855258cef621a0942875910e416 parent 0e4b407d02da9f565a5ba6def26db22565925cbd Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:13:52 +0100 bump LICENSE to 2020

cleanup includes and remove unused fields from struct feed

2020-02-01T14:13:33Z

commit 0e4b407d02da9f565a5ba6def26db22565925cbd parent c0ae8d7b1f540d5cda35df1abe323aecf8ab3ff5 Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:13:33 +0100 cleanup includes and remove unused fields from struct feed

style.css: sort properties

2020-02-01T14:09:38Z

commit c0ae8d7b1f540d5cda35df1abe323aecf8ab3ff5 parent 8726a24365b8f746bcbb48a907a90afc8031b2f2 Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:09:38 +0100 style.css: sort properties

README: small change

2020-02-01T14:05:02Z

commit 8726a24365b8f746bcbb48a907a90afc8031b2f2 parent e24125d0f7679e0612c7ed69d79f07654f10637d Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:05:02 +0100 README: small change

Makefile: simplify Makefile, rm config.mk, use system cflags/ldflags

2020-02-01T14:03:00Z

commit e24125d0f7679e0612c7ed69d79f07654f10637d parent 5df58d27f557292778cdc5dee306f18db8c980f7 Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:03:00 +0100 Makefile: simplify Makefile, rm config.mk, use system cflags/ldflags

sync XML improvements

2020-02-01T14:02:27Z

commit 5df58d27f557292778cdc5dee306f18db8c980f7 parent f8629e681a16fc3af086355a44c942df57291b4b Author: Hiltjo Posthuma Date: Sat, 1 Feb 2020 15:02:27 +0100 sync XML improvements

bump version to 0.3

2019-08-18T15:06:09Z

commit f8629e681a16fc3af086355a44c942df57291b4b parent 2bc2be55ea8cc026fc822ee8031dfd49bb77d7cc Author: Hiltjo Posthuma Date: Sun, 18 Aug 2019 17:06:09 +0200 bump version to 0.3

man pages: fix mandoc lint warnings + wording tweaks

2019-08-18T12:01:29Z

commit 2bc2be55ea8cc026fc822ee8031dfd49bb77d7cc parent f05f3eb6c90f7b1baf7369498609dc5d5d212b63 Author: Hiltjo Posthuma Date: Sun, 18 Aug 2019 14:01:29 +0200 man pages: fix mandoc lint warnings + wording tweaks

man page documentation for all tools: copied from sfeed and changed

2019-08-17T10:10:00Z

commit f05f3eb6c90f7b1baf7369498609dc5d5d212b63 parent 797f715398aeac08febc8067ed6423da727e4f45 Author: Hiltjo Posthuma Date: Sat, 17 Aug 2019 12:10:00 +0200 man page documentation for all tools: copied from sfeed and changed initial version, needs some more work.

xml: improve cdata and comment callback logic

2019-08-02T16:46:06Z

commit 797f715398aeac08febc8067ed6423da727e4f45 parent 51995d6fc4760fadac68650bb82773b9bf9eae79 Author: Hiltjo Posthuma Date: Fri, 2 Aug 2019 18:46:06 +0200 xml: improve cdata and comment callback logic it used to call both handlers twice at the end for "-->" (comment) or "]]>" (CDATA) with the data "" and length 0. Now it is only called when non-empty. The start and end handlers can still be used.

tscrape_update: sync improvements from sfeed_update

2019-08-02T16:33:10Z

commit 51995d6fc4760fadac68650bb82773b9bf9eae79 parent db47c97bea3370886d011a2c950ead2551cf3fbc Author: Hiltjo Posthuma Date: Fri, 2 Aug 2019 18:33:10 +0200 tscrape_update: sync improvements from sfeed_update - change order of functions in script and documentation to match the execution order. - improve a comment about the parallel processing behaviour (performance stall).

tscrape_update improvements

2019-05-12T17:20:49Z

commit db47c97bea3370886d011a2c950ead2551cf3fbc parent 5e6e62cf3522747a7c4573736d774503ff139a12 Author: Hiltjo Posthuma Date: Sun, 12 May 2019 19:20:49 +0200 tscrape_update improvements - Better checking and verbose logging (on failure) of each stage: fetchfeed, filter, merge, order, convertencoding. This makes sure on out-of-memory, disk-space or other resource limits the output is not corrupted. - This also has the added advantage it runs less processes (piped) at the same time. - Clear previous unneeded file to preserve space in /tmp (/tmp is often mounted as mfs/tmpfs). - Rename fetchfeed to fetch. - Add logging function (able to override), use more logical logging format (pun intended). - Code-style: order overridable functions in execution order.

README: add preface text, list dependencies

2019-05-12T17:19:37Z

commit 5e6e62cf3522747a7c4573736d774503ff139a12 parent 2f683439be0f889e05e42965926d44a7332f042d Author: Hiltjo Posthuma Date: Sun, 12 May 2019 19:19:37 +0200 README: add preface text, list dependencies

example: fix typo in name

2019-05-12T17:19:29Z

commit 2f683439be0f889e05e42965926d44a7332f042d parent fb64d1d7eb24caab8ca7fb574ffad5886ff8f05f Author: Hiltjo Posthuma Date: Sun, 12 May 2019 19:19:29 +0200 example: fix typo in name

tscrape_update: disable If-Modified-Since by default

2019-05-12T16:58:37Z

commit fb64d1d7eb24caab8ca7fb574ffad5886ff8f05f parent 0adbb9ec4d4f8e92643b1dfc737c1780a1c4c7a3 Author: Hiltjo Posthuma Date: Sun, 12 May 2019 18:58:37 +0200 tscrape_update: disable If-Modified-Since by default

bump version to 0.2

2019-04-23T19:07:24Z

commit 0adbb9ec4d4f8e92643b1dfc737c1780a1c4c7a3 parent 33e40c9fc70a418c65d6de3cc68c1c3431f4cd47 Author: Hiltjo Posthuma Date: Tue, 23 Apr 2019 21:07:24 +0200 bump version to 0.2

Makefile: make it simpler to not compile compat objects

2019-04-23T19:06:48Z

commit 33e40c9fc70a418c65d6de3cc68c1c3431f4cd47 parent f5f136fabf5f8cd7245d963e7e479442200738e7 Author: Hiltjo Posthuma Date: Tue, 23 Apr 2019 21:06:48 +0200 Makefile: make it simpler to not compile compat objects on OpenBSD: make COMPATOBJ=

util.h: remove unused macro ISUTF8

2019-04-23T19:06:31Z

commit f5f136fabf5f8cd7245d963e7e479442200738e7 parent 8af4c93ad7dee6454abd9da106b9f3e37e1ce400 Author: Hiltjo Posthuma Date: Tue, 23 Apr 2019 21:06:31 +0200 util.h: remove unused macro ISUTF8

README: update, add twitter to Atom example

2019-04-22T12:46:41Z

commit 8af4c93ad7dee6454abd9da106b9f3e37e1ce400 parent 2872a29d4f44afbfa4f439ba1f3d84c22114b0d4 Author: Hiltjo Posthuma Date: Mon, 22 Apr 2019 14:46:41 +0200 README: update, add twitter to Atom example

sync XML improvements

2019-04-22T12:46:30Z

commit 2872a29d4f44afbfa4f439ba1f3d84c22114b0d4 parent bd299de160e8f56d6f88538d9d4d4ded4775038d Author: Hiltjo Posthuma Date: Mon, 22 Apr 2019 14:46:30 +0200 sync XML improvements

Makefile / make dist: there is no TODO file

2019-02-22T12:11:48Z

commit bd299de160e8f56d6f88538d9d4d4ded4775038d parent 0743e76986d211f2a2695410c7678275355341d0 Author: Hiltjo Posthuma Date: Fri, 22 Feb 2019 13:11:48 +0100 Makefile / make dist: there is no TODO file

Makefile: add DOCPREFIX for installing docs in ports, use system {C,LD}FLAGS

2019-02-22T12:09:12Z

commit 0743e76986d211f2a2695410c7678275355341d0 parent 1e8d328abcab552a8a58fee93eadada8b42148f7 Author: Hiltjo Posthuma Date: Fri, 22 Feb 2019 13:09:12 +0100 Makefile: add DOCPREFIX for installing docs in ports, use system {C,LD}FLAGS change installed doc from /usr/local/share/tscrape to /usr/local/share/doc/tscrape

xml: remove unnecesary checks

2019-02-22T12:02:22Z

commit 1e8d328abcab552a8a58fee93eadada8b42148f7 parent b11135a350c346d434708139583668e95b96427f Author: Hiltjo Posthuma Date: Fri, 22 Feb 2019 13:02:22 +0100 xml: remove unnecesary checks - reduce amount of data to check. - remove unnecesary checks from (now) internal functions.

rename getchar_ignore to getnext_ignore

2018-12-18T17:10:33Z

commit b11135a350c346d434708139583668e95b96427f parent 36af5420b34500f32a5a8d3ee6601e57f3619bf8 Author: Hiltjo Posthuma Date: Tue, 18 Dec 2018 18:10:33 +0100 rename getchar_ignore to getnext_ignore

tscrape.1: formatted timestamp field was removed a long time ago

2018-12-17T17:55:32Z

commit 36af5420b34500f32a5a8d3ee6601e57f3619bf8 parent 0368be87df8da32cab27d2d5fab28c053d352016 Author: Hiltjo Posthuma Date: Mon, 17 Dec 2018 18:55:32 +0100 tscrape.1: formatted timestamp field was removed a long time ago

tscrape_update: remove unused terminated() function

2018-12-17T17:40:28Z

commit 0368be87df8da32cab27d2d5fab28c053d352016 parent ed3a979265abe557e783ea22c6a09fb96241ff95 Author: Hiltjo Posthuma Date: Mon, 17 Dec 2018 18:40:28 +0100 tscrape_update: remove unused terminated() function

ignore incorrect unescaped HTML in

tscrape, branch HEAD

README: small improvement

README: fix twitter to Atom (sfeed) example

bump LICENSE year

update static bearer token

bump version to 0.6

tscrape: remove debugging comments

tscraperc.example: change feeds

tscrape_update: escape { and } in sed expression

tscrape_update: increase timeline limit from 50 to 100

refactor urls into general replacement function and replace some HTML entities

README: some rewording

rm code thats not needed anymore and a debug/test comment

Expand all photos URLs

Expand extended_entities URLs in retweets

Use extended_entities instead of entities

Further expand URLs in retweets

update man pages a bit for the recent changes

replace all whitespace by a single space

Fix an off-by-one in printexpand() by leot

simplify parsejson error checking

replace newline to space instead of removing them

work-in-progress: support the new Twitter site

fix typo

bump version to 0.5

style.css: improve horizontal scrolling for long titles/small windows

util: printescape(): it's enough to set a flag, no counter needed

documentation: no need to explicitly set HTTP/1

tscrape_update: tiny style fix in comment

tscrape_html: use a section per feed

when it is inside . Then it ignores newlines (inside ) and the links are inlined. - w3m does not have a (line)margin for h2.

man page improvements (sync)

tscrape_html/tscrape_plain: like sfeed, skip when a timestamp is empty/invalid

README: don't set LC_ALL for awk example

sync printutf8pad from sfeed

ignore possible consecutive `js-stream-item' when parsing a single tweet

bump version to 0.4

tscrape_update: don't preserve permissions of tmp files by moving, so copy

cleanup some more includes

rm example.sh

bump LICENSE to 2020

cleanup includes and remove unused fields from struct feed

style.css: sort properties

README: small change

Makefile: simplify Makefile, rm config.mk, use system cflags/ldflags

sync XML improvements

bump version to 0.3

man pages: fix mandoc lint warnings + wording tweaks

man page documentation for all tools: copied from sfeed and changed

xml: improve cdata and comment callback logic

tscrape_update: sync improvements from sfeed_update

tscrape_update improvements

README: add preface text, list dependencies

example: fix typo in name

tscrape_update: disable If-Modified-Since by default

bump version to 0.2

Makefile: make it simpler to not compile compat objects

util.h: remove unused macro ISUTF8

README: update, add twitter to Atom example

sync XML improvements

Makefile / make dist: there is no TODO file

Makefile: add DOCPREFIX for installing docs in ports, use system {C,LD}FLAGS

xml: remove unnecesary checks

rename getchar_ignore to getnext_ignore

tscrape.1: formatted timestamp field was removed a long time ago

tscrape_update: remove unused terminated() function

tscrape_html: use a
section per feed

when it is inside
. Then it ignores newlines (inside
) and the links are inlined. - w3m does not have a (line)margin for h2.