00:00:00 --- log: started forth/18.10.04 01:10:49 --- join: ncv (~neceve@2a02:c7d:c5c9:a900:6eaf:6ef7:3b81:d5f6) joined #forth 01:10:49 --- quit: ncv (Changing host) 01:10:49 --- join: ncv (~neceve@unaffiliated/neceve) joined #forth 02:00:27 rdrop-exit: the BASIC on my old Sharp PC1500A did that. after entering a line it was tokenized and stored as a stream of tokens rather than characters 02:01:12 This is not the same thing, youâre tokenizing source. 02:02:47 --- quit: proteus-guy (Ping timeout: 260 seconds) 02:03:15 --- quit: ashirase (Ping timeout: 264 seconds) 02:05:20 In a sourceless environment you would not be storing the source, you would store object code and just enough extra information to aid disassembly. 02:05:22 --- join: proteus-guy (~proteusgu@2403:6200:88a6:329f:2920:1ca0:b94d:d9b) joined #forth 02:09:02 --- join: ashirase (~ashirase@modemcable098.166-22-96.mc.videotron.ca) joined #forth 02:10:14 --- quit: nighty- (Quit: Disappears in a puff of smoke) 02:13:13 At least that is what I understood from Chuckâs interview. 02:13:32 It reminds me a lot of the M-Code cross-assembler/disassemler. 02:14:27 --- quit: proteus-guy (Ping timeout: 260 seconds) 02:15:43 Itâs a more extreme approach than ColorForth. 02:16:48 --- join: proteus-guy (~proteusgu@2403:6200:88a6:329f:2920:1ca0:b94d:d9b) joined #forth 02:17:12 He discusses it in the last few minutes of the interview. 02:17:26 --- join: john_metcalf (~digital_w@host109-149-153-210.range109-149.btcentralplus.com) joined #forth 02:17:38 Iâm hoping heâll give further details at the upcoming SVFIG Forth Day. 02:42:07 --- quit: Labu (Quit: WeeChat 2.0.1) 02:43:28 --- join: Labu (~Labu@labu.pck.nerim.net) joined #forth 03:00:01 --- join: nighty- (~nighty@s229123.ppp.asahi-net.or.jp) joined #forth 03:38:46 rdrop-exit: I've spent a lot of time thinking about that part of Moore's thinking. 03:39:58 As far as I could tell from some of the older stuff about that in the Ultratechnology website, he has a persistent symbol table. What he stores in his "object code" is a pointer into that symbol table. There's a field in the symbol table that supplies the word's CFA. If the word hasn't been compiled yet, that field is zero. 03:40:13 When he compiles it, he sets that CFA field in the symbol table. 03:40:59 --- nick: Guest50250 -> KipIngram 03:41:08 --- mode: ChanServ set +v KipIngram 03:41:39 So when he's compiling a definition he just has to follow the object pointer to the symbol table, grab the CFA, and shove it into memory. 03:42:32 No search. 03:43:18 You can create effective compression schemes such that the most frequently used words (most frequently used indices into the symbol table) can be stored compactly in the source. 03:44:11 I decided this had a limitation. If the symbol table can supply only one CFA value for any given entry, that implies that a given symbol string can have one and only one meaning, throughout your system. 03:44:25 That conflicts with vocabularies, where you can define the same string to mean multiple things. 03:45:08 You could try to overcome that by "complexifying" the symbol table - try to have it so it was the vocabularly path + symbol string that led to a symbol table entry. 03:45:24 But in my case I have multiple processes, and so processes can multiply define symbols too. 03:45:28 So I'd also have that to deal with. 03:46:44 Youâre speaking of ColorForth, sourceless Forth which was a different approach he was using in his cad system before CF, that he had given up on but is now revisiting. 03:46:57 I opted to deal with it by having my symbol table simply give me a string <--> unsigned integer mapping, and then I still have traditional linked header lists in each vocabulary / process that works in the usual way, except they store the integer instead of the string. 03:47:37 So I don't really consider my system "sourceless" in any way - I just consider it as supporting source compression. 03:48:22 And it likely will speed up compilation *some*, since I will only have to do integer comparisons when searching header lists. 03:49:21 I could probbaly play some trickery and have a spot in the symbol table to store one CFA and whether or not a symbol was only defined once in the system. 03:49:35 If so, I could use that CFA slot - if not, I'd have to search local dictionaries. 03:49:47 Then I'd get the benefit for any symbols that were not multiply defined. 03:51:28 Or something along those lines - I haven't really thought enough about it. 03:51:55 The source tokens could say "use symbol table CFA" or "use local dictionary." 03:54:38 Actually that wouldn't work, would it? It would imply an order in which source was loaded. 03:54:51 Once again, youâre speaking of an approach similar to CF which tokenizes/compresses source code, this is not the same approach as a Sourceless Forth. 03:55:03 Anyway, I abandoned my attempts to figure out how to extend that idea to multiple meanings. 03:56:03 Youâre speaking of an approach similar to ColorForth, sourceless Forth was a different approach he was using in his cad system before CF, that he had given up on but is now revisiting. 03:56:12 Well, the way he did it feels like it's a huge step toward sourceless - the stored form o the program more or less supplies the meaning independent of other structures. 03:56:20 I mean, the names have to be stored somewhere, right? 03:56:35 Or is he hashing them somehow? 03:58:07 And since hashes are one way, how would you ever get a rendering of your source in an editor? 03:58:43 I don't see how to do this without a symbol table, and if you're able to store everything you need to compile IN the symbol table, that seems pretty sourceless to me. 04:02:50 When you want to view the source, you recover the source by disassembling the object with the aid of supplemental info (opcode table, labels, comments). You store the object code and just enough extra info apart from the object code to reproduce the source. 04:03:07 --- join: dddddd (~dddddd@unaffiliated/dddddd) joined #forth 04:04:40 Yes, that describes how this would work too. You recover the source by using the tokens to access the symbol table, so that you can render the names. 04:05:08 I do see how some traditional Forth words might present problems there. 04:05:10 The source is not tokenized. 04:05:27 Like IF ... THEN, which doesn't get compiled in quite that way. 04:06:33 Ok, so give me an example. 04:07:01 What is stored for some arbitrary word foo. 04:08:19 And how do you recover "foo" from that when you're decompiling? 04:08:40 I'm just trying to glom on to exactly how this is different from tokenized. 04:11:25 A symbol table entry, that points somewhere in the object code. 04:12:21 Does the object code point back? 04:12:35 Or do you have to search the symbol table for the right entry when decompiling? 04:13:23 No the object code is machine instructions for the processor, itâs not an intermediate form. 04:13:45 Ok. I'm starting to see the difference then. 04:13:54 So you would have to search the symbol table. 04:14:03 This wouldn't be as fast as his ColorForth era stuff, then. 04:14:41 But who cares - he isn't actually "compiling," so I guess in that sense it's "faster." 04:14:46 Since he's eliminating that phase altogether. 04:14:57 Those symbol table searches would occur at edit time, right? 04:15:09 Right 04:15:26 Ok, thanks - I think I get the difference now. Interesting. 04:17:06 So, regarding structures like IF ... THEN --> ?jmp, anything interesting to say about how he knows how to decompile that back to IF ... THEN? 04:17:18 Or did he change his whole language structure to accomodate this? 04:17:31 Maybe he just eliminated such things. 04:17:40 When thinking about what Chuck ideas of Forth, I think itâs useful to think of it as assembly programming (sometimes for a real processor, sometimes for a virtual processor) rather than thinking of Forth as a compiled or interpreted language. 04:18:14 I guess that last question is "Did he change the language to eliminate decompilation ambiguity?" 04:18:41 Chuck tries to minimize the semantic gap between source code and object code, between language and processor. This is what he means in the video when he says "one-to-one correspondence between source and object". In effect his development environment would be closer to an interactive assembler/disassembler than a traditional compiler or interpreter for a parsed language. 04:19:02 Ok, I'll take that as "yes." 04:19:17 I'll assume he meant "one to one" in the mathematical sense, which means "no ambiguity." 04:19:42 He did not give details, Iâm hoping heâll explain more at the SVFIG Forth Day later this month. 04:20:16 Have a look at the last few minutes of his EuroForth 2018 interview. 04:20:24 K. 04:20:58 That one-to-one requirement restricts the language to some extent. 04:21:07 In regular Forth nothing is stopping me from doing this: 04:21:15 https://wiki.forth-ev.de/doku.php/events:ef2018:start 04:21:18 : foo postpone bar ; immediate 04:21:25 : bam postpone bar ; immediate 04:21:33 Now foo and bam compile to exactly the same thing. 04:21:37 That's not one-to-one. 04:22:00 Now, you could ask why I'd want to do that, and it would be a fair question. 04:22:13 In my Forth that would be Â«Â : foo bar ;inlineÂ Â» 04:22:25 But the answer might be that some days I just feel foo and some days bam. 04:22:55 If I'm not allowed to do it, that's a restriction. 04:23:40 Numerous things in Forth wind up compiling ?jmp or jmp. 04:24:11 Maybe it's possible to look at the local object structure to back-infer what control structure was in the source. 04:26:04 So when you compile a high level Forth word, you get just a list of addresses. 04:26:25 Since you can load source in varying order, the same source can wind up having different addresses on different loads. 04:26:35 : foo a b c d ; 04:26:52 On one load, you could have a > b > c > d, on another some other ordering, depending on how you loaded prior source. 04:27:02 How does this new approach handle unloaded code? 04:27:30 Unloaded code? 04:27:44 Something you'd defined but hadn't loaded from disk to RAM yet. 04:28:06 Seems there has to be a representation of foo that is not based on RAM addresses. 04:29:19 I could see it perhaps being done by regarding the disk space this is all stored on as a "virtual address space," and then everything would be "compiled there." 04:29:37 Not necessarily, if youâre interactively and assembling you always have object code. 04:29:39 Then when you brought it into RAM you'd have to do some sort of conversion, so that the *loaded* definitions now used RAM addresses. 04:30:26 I could see that all being resolved with a virtual memory system. 04:30:49 It would all be built in "virtual space," and the fact that you had some of it in physical RAM would really be irrelevant. 04:32:04 I donât think Chuck loads things piecemeal from disk anymore, just the development environment as a whole. I recall him saying his block was just Â«Â : block 1024 * ; Â«Â 04:32:38 Aha - yes - that smacks of a big virtual address space. 04:32:41 Makes sense. 04:33:05 Thatâs the way my Forth works. 04:33:21 And I had definitely gotten the impression that his later stuff has moved away from being "generic" in the same way old Forth was, and is more about building something that precisely targets a particular application. 04:33:52 My block is defined as Â«Â : block ( blk# -- a ) k STORAGE + ;inlineÂ Â» 04:34:06 That's in keeping with his long-term philosophy: Solve the problem in front of you; waste no energy "supporting the solution of problems you don't have." 04:35:28 Right 04:35:45 I lusted after his CF era "fast compile" pretty badly. 04:36:01 But in the end I decided it wasn't entirely compatible with what I wanted to be able to do with my system. 04:36:06 I want an *operating system*. 04:36:16 Time for dinner, gotta go, nice chatting with you. 04:36:18 That's inherently a system that I might want to solve problem A with right now, and B tomorrow. 04:36:30 Or A and B today, B and C tomorrow, A and C the next day. 04:36:53 I felt like I had to have vocabularies, and had to have multiple processes able to define the same string to mean different things. 04:37:08 So I wasn't focused in the same way CM was in that era. 04:37:18 I think around then he was focused on being able to compile chip designs. 04:37:30 And his "development system" would be devoted to that one thing. 04:37:50 Obviously that specificity could be exploited. 04:39:00 The whole notion of "operating system" is sort of contrary to CM's view of things. 04:39:11 It's kind of automatically excess baggage. 04:40:38 Of course with the "virtual memory" way of looking at it you could try to have both things. Each application would be its own totally separate virtual space. 04:40:50 And that might be something modern processors could handle quite well. 04:42:01 What happened to forth.org and the ultratechnology.com site? 04:42:02 I have noticed in my MacOS Forth the thing gets its own virtual space on the Mac every time it's loaded. 04:42:08 Independent of what else I may be doing with the computer. 04:42:25 Everything's always at the same address, unless I add code such that it moves. 04:42:40 The ultratech site is still "there." 04:42:45 I pulled it up a couple of days ago. 04:42:55 I don't think it reflects any current activity. 04:43:03 Just a bunch of historically interesting documents and videos. 04:43:28 It's not still "there". 04:43:43 http://www.ultratechnology.com/ 04:43:50 Just opened it just now. 04:44:04 Are you not getting a 403? 04:44:09 No, getting a page. 04:44:22 "UltraTechnology has been out of business since 2002. This site documents its history with code, simulations,..." 04:45:05 No idea what the hell is happening on my end then 04:45:48 Hmmm. I can't ping the site. 04:46:04 Is it possible it's gone down recently and I had something in a cache or something? 04:46:14 Let me try it from Safari, where I've never previously opened it. 04:46:50 It opens there too. 04:47:02 That's weird. 04:48:34 So anyway, when I load my system MacOS sets it up so that everything is always at the same virtual address. So I certainly could use some kind of sourceless representation that just had pointers into that compiled code. 04:49:31 My language doesn't have / maintain that 1-to-1 correspondence, though. 04:51:00 Strictly speaking, though, I don't know that this needs to be *completely* 1-to-1. I could have process A define : foo bar ; and process B define : foo bam ; 04:51:16 foo would be in the symbol table twice, pointing at bar once and bam once. 04:51:40 If you search for the symbol table entry that points to bar, you find foo. 04:51:43 Similarly for bam. 04:51:48 So decompilation works right. 04:52:05 You'd still have to have some way of knowing, when you typed foo and hit enter, which one to run. 04:52:20 But that's an "edit time / interaction time" thing, and there could be structures for that. 04:53:01 Structures that were accessed only when you were editing your code; "your foo" would get stored in the code pointing to the right place. 04:53:43 As far as decompilation and rendering goes, this could be a many-to-one mapping. 04:54:55 Something as simple as having the symbol table be a collection of linked lists would work - different processes would have different "entry points," and within that linked list it would be 1-to-1. 04:55:09 mmm, dat sexy html1.0 04:55:28 Oh, the UltraTech site? Yeah - pretty "primitive" looking. :-) 04:55:42 For someone my age it at least has some nostalgia. 04:56:15 Things were more... "pragmatic" in those days. 04:56:20 Deliver the information - period. 04:59:30 I guess I think the later evolution of web presentation is fine - as processors and network tech grew in power, we became able to be "prettier." 04:59:45 The part I object to, hugely, is the decisions that led to the ability for websites to infect your computer and so on. 05:00:00 It should have been kept focused on rendering stuff on your screen. 05:00:12 This whole "active content" business was a bad path. 05:00:30 Some really bad security decisions were made. 05:02:14 I don't know how they botched it so badly - the *concept* of your browser providing a virtual computing sandbox that web content could "run on" isn't broken in and of itself. 05:02:32 The idea of cookies isn't bad in and of itself. But they managed to fail in the implementation somehow. 05:05:37 rdrop-exit: I've thought at times about a better 1-1 mapping of source and compiled code. 05:06:42 Instead of IF ... THEN implemented with ?jump you could have ?SKIP: . That gets you the same capabilities, and encourages (forces) factoring - the ... content would be in its own word. 05:07:00 And it's immediately decompile-able. 05:07:22 Doesn't require immediacy. 05:08:08 Another way of doing the same thing would be to put a conditional return at the front end of the ... word. 05:08:15 I've done that a good bit in my code. 05:08:31 I find that with my conditional returns I use a lot fewer IF ... THEN clauses. 05:08:52 ?SKIP: would be a little faster, since it would avoid the call/return. 05:09:51 But if everything has to be compiled into using jump and ?jump then it's harder to avoid ambiguous stuff. 05:11:22 --- join: siraben (~user@unaffiliated/siraben) joined #forth 05:15:40 Do things like race conditions appear at all at the assembly level or Forth? I never encountered them anywhere. Say I wanted to make a race condition? 05:17:28 Or do such things only happen within the context of threading and OSes? 05:17:47 if you have two independent workers then simply put variable incrementation in loop and see, if it sums up when both workers are done 05:17:48 That kind of thing comes up anytime you have multiple threads. 05:18:02 Have you implemented threads? 05:18:23 Well, you need either multiple cores, or else a mechanism for interrupting one thread and switching to another context. 05:18:25 I can't see how interleaving computations can make them faster than doing one at a time. 05:18:31 Oh, I don't have multiple cores, that's why. 05:18:37 It doesn't, really. 05:18:42 Unless you have multiple cores. 05:18:51 unless you have blocking operations 05:18:53 You can just provide the impression to the user that multiple things are happening at once. 05:19:08 But the time you spend switching back and forth is "wasted." 05:19:18 I have a non-blocking version of KEY called KEYC which returns 0 when no key is entered 05:19:33 Right - you could build multi-threading around that. 05:19:35 By the way, not sure if that's a good name for it, suggestions? 05:19:53 read-char-no-hang ? 05:19:54 Application would use KEY but under the hood it would check to see if a keystroke is available, and if it's not go do something else for a little while. 05:20:10 I usually call it ?KEY. 05:20:15 The one that checks and returns a flag. 05:20:16 So what does ? mean? 05:20:24 maybe-key 05:20:34 Returns a flag? 05:20:37 Yeah, or "check key" or whatever. 05:20:41 I don't have any flags, hm. 05:20:54 ?KEY will return 0 or 1 - not the actual keystroke. 05:21:00 Huh. 05:21:02 You'd still have to run KEY to get it. 05:21:11 But you could avoid having KEY block. 05:21:14 So how would you use it? 05:21:32 ?KEY IF KEY THEN 05:21:35 would be the simplest way. 05:21:53 For me it's BEGIN KEY DUP 9 <> WHILE CASE 1 OF FOO ENDOF 2 OF BAR ENDOF ... ENDCASE REPEAT DROP 05:21:55 But you could also say ?KEY IF KEY ELSE ...run something else... THEN 05:22:00 I mean KEYC, not key^ 05:22:03 In that example 05:22:08 Oh I see. 05:22:22 I thought ?KEY had this stack image ( -- F | keycode T ) 05:22:35 Or just run KEYC until a non-zero value is returned in my case. 05:22:38 You could put it in a loop, and the loop body would do other useful work while waiting for a key to be ready. 05:22:45 So what are flags? 05:22:51 The processors flags themselves? 05:22:54 Just a 0 or 1 returned on the stack. 05:22:59 No, I meant a "Forth flag." 05:23:05 A Boolean. 05:23:09 Ah I see. 05:23:27 in some forths for true you do not get 1 but 0xFFFF (whatever your cell size) 05:23:28 I'm using the convention of 0 = false 1 = true, but apparently it's supposed to be 65535 (or all 1s) 05:23:36 For a while Chuck Moore had IF use the processor flag, but he later went back to stack "flags," I think. 05:23:47 It can be whatever you want it to be. 05:23:52 Of course. 05:23:55 My last one used -1, this current one uses 1. 05:24:00 Zarutian: interesting 05:24:11 What does your STATE = 0 correspond to? 05:24:17 Interpreting. 05:24:23 STATE = 1 is compiling. 05:24:31 Oh, I got it the other way round. Hm. 05:24:39 :-) Convention. 05:24:47 Be consistent and you're golden. 05:24:52 the handy thing of having true being 0xFFFF is that AND, OR, XOR and other boolean and multibit operations are the same 05:25:00 Right. 05:25:07 Ah, that's why. 05:25:19 In a few places I couldn't resist the urge to "do math" on my true/false results. 05:25:29 But that can work either way - you just have to do the right math. 05:25:35 I'm thinking of adding 32 bit numbers and operations but can't possibly see how they would be fast. 05:25:53 Surely there's a collection of 32 bit snippets for the Z80 somewhere 05:25:53 Doubles really aren't fast if you have to implement them on top of singles. 05:26:21 I decided that 64 bits was wide enough; I passed over implementing doubles. 05:26:33 Of course. 05:26:50 I use decimal points in numbers to trigger floating point conversion. 05:27:04 Back from dinner 05:27:15 Interesting fact: the TI 84 calculator OS can do up to Â± 2^33 calculations exactly, more than that it uses floating point. 05:27:32 But it has rationals, reals, complex, etc. because it's a calculator. 05:27:51 Yeah, rational computing is pretty interesting. 05:27:53 Considering that you can barely multiply on the Z80, that's pretty impressive. 05:28:16 crappy IEEE 754 floating points are the bane of my existance, gimme scaled values instead when the range of values are known 05:28:24 It's exact. There's a mathematician out there named Wildberger or something like that that has developed a complete theory of math (or a lot of it, anyway) using rationals. 05:28:33 He has a well-developed theory of trigonometry. 05:28:35 I use two calculators HP16C and HP48GX 05:28:35 Should I use BCD? 05:28:50 For floating point? 05:28:55 For doubles 05:29:00 Or arbitrary precision 05:29:05 Oh, I wouldn't. That will be slower still. 05:29:20 Even for arbitrary precision, I'd still use full cells for the hpieces. 05:29:23 KipIngram: rationals? sorry my English vocabulary isnt good mathwise. It is the x / y things? Like one learned about in elementary school. 05:29:24 It will be faster. 05:29:25 I don't trust my asm-fu enough to do 32 bit multiplications 05:29:28 Zarutian: Yes. 05:29:31 "Fractions" 05:29:39 Yes, rational number is quotient of two integers. 05:30:14 right. Now I know you are talking about. 05:30:22 pi Convergent rational approximation of pi. 05:30:23 : pi ( -- numerator denominator ) 05:30:24 $ 24baf15fe1658f99 $ bb10cb777fb8137 ; 05:30:39 siraben: a 16-bit cell holds roughly three decimal digits. 05:30:50 If you go BCD, you will have to do many more operations. 05:30:57 *gasp* 05:31:07 Since you'll be working with one digit at a time, when you could be handling three-ish. 05:31:33 been thinking of stealing ideas from Scheme's (the Lisp) numerical tower system. 05:31:48 Ah yes, that's quite good. 05:31:59 rdrop-exit: Wow - how many digits does that get right? 05:32:06 Common Lisp has a similar numerical tower 05:32:19 Zarutian: So you're going to support fractions, complex numbers, reals, and signed versions of each? 05:32:44 I'm actually thinking of support for tensors. 05:32:53 Not in the core system, but as a "library" of sorts. 05:33:05 I just wished that a spefic eletronics CAD and simulation package used CL or Schemes numerical tower when simulating circuits and not just LTSPICEs shitty floating point based implementation. 05:33:06 Thatâs the best rational approximation for a 64-bit u*/ with a 128-bit intemediary product 05:33:42 Donât remember how many digits it gets right 05:34:57 I've seen a mandelbrot program in Forth 05:35:42 Nice. 05:36:19 Oh thank goodness: http://z80-heaven.wikidot.com/math 05:37:00 Do people use Forth for cryptographic applications? 05:37:06 Vectors, matrices, and tensors always have components, and those components only have a well-defined meaning in the context of a specified reference frame. So that stuff I mentioned above would store not only the components, but also a reference frame indication. 05:37:15 Then it can be thought of as representing something physically real. 05:37:40 And the underlying code could switch it between coordinate systems, etc. 05:38:07 Maybe at some point I should just turn this interpreter into a larger system and replace the TI OS 05:38:25 Yeah, why not? 05:38:44 Except I'd suddenly make my calculator not a calculator 05:38:50 Then you could get rid of that flash-app annoyance you were discussing yesterday. 05:38:59 Well, unless you made it back into one. 05:39:26 Right. It's just a bit tedious figuring out using these ROM calls 05:39:26 I.e., had you own calculator functionality. 05:40:05 There seems to be a little architecture for the OS. There's 6 9-byte floating point "registers" with specific calls to transfer data from one to another 05:40:21 Actually, you might be interested to know that OS variables are type-tagged. 05:46:39 TI produced lisp machines in 80, it is not suprising they have retained some ideas 05:47:19 Do they have a compiler for internal use or do you think that their engineers just write in ASM? 05:48:33 Computing would be different if the Lisp machines won. 05:49:13 sure, enterprise computing all along the way ,) I'm happy with Common Lisp on stock hardware 05:50:10 Forth is like Lisp in many ways, especially with respect to its meta-programming capabilities. 05:59:59 I poked into that not long ago, and got the notion that that similarity is somewhat superficial. 06:00:06 Under the hood I think they're pretty different. 06:00:49 Forth is the "ultimate" low-level language, Lisp is the "ultimate" high-level language, I think. 06:02:11 --- quit: rdrop-exit (Quit: rdrop-exit) 06:10:40 What should I call a word that allocates the number of bytes given by the top of the stack? 06:10:53 Like ALLOT but more permanent because the data resides outside the Forth system. 06:11:00 ALLOC? 06:30:08 That seems reasonable, but the important thing is that it has meaning to you. 06:30:49 My "outside the system" allocation is from my heap - I called those words HEAP and -HEAP (allocate a page, free a page). 06:32:29 I actually have two layers of that - (HEAP) and (-HEAP) do a "raw" and very fast allocation / deallocation; HEAP and -HEAP use the first two cells of the page to maintain a doubly linked list of all of a process's heap pages and return the address just past those cells to the user. 06:32:59 --- join: mark4 (~mark4@cpe-2606-A000-8096-FE00-5A94-6BFF-FEA6-29D4.dyn6.twc.com) joined #forth 06:33:40 Error recovery follows those links and snapshots every page belonging to the process for possible restoration if an error is encountered. 06:34:52 In addition to user-code calls to HEAP / -HEAP, pages get automatically allocated now and then as needed for dictionary growth. 06:35:33 I think I want error recovery to "unwind" any user calls to HEAP or -HEAP that occurred on the error-containing line. 06:35:50 Otherwise pages can get allocated and then error recovery destroys any refernece to them - it's a leak. 06:35:54 I see. 06:36:02 Unwinding HEAP calls is no problem; I can add that easily. 06:36:06 The TI-84 has a Variable Allocation Table so nothing is really leaked. 06:36:16 Just de-allocated by a ROM call. 06:36:27 Each variable is identified by a type and a 8-character name. 06:36:34 Unwinding -HEAP calls is easy too, but if I have multiple threads running the possibility exists that another thread could allocate the freed page before it gets "un-freed." 06:36:39 So that's a glaring problem. 06:36:50 Ah, free issues. 06:36:54 I let the OS do that. 06:37:12 This is like a tagged malloc 06:37:21 I could just mark the need to free the page but postpone the actual freeing until successful completion of the line, but then I've got a problem if I run a word that does a lot of heap churning and runs for a long time. 06:37:28 Well, this is meant to be the OS. 06:37:35 It's not, yet, but it will be. 06:38:28 The way I have this written right now, HEAP will return the most recently freed page preferentially. 06:38:38 So that's just a problem waiting to happen when I go multi-thread. 06:42:01 I could reduce its severity by deferring free for up to some small number of pages. 06:42:23 Then in most situations the frees would get deferred to success, but if a word really took off and needed to do a lot of stuff it would be able to. 06:42:43 Imperfect, but might cover me "well enough." 06:43:10 What do you mean by a page? 06:43:18 Is that one unit of allocation? 06:43:23 Or OS page? 06:43:34 Yes - to keep this efficient I allocate fixed size pages from this "heap." 06:43:53 How do you deal with address space layout randomization? 06:44:05 Not sure what you mean. 06:44:17 Modern OSes tend to randomize their memory allocation 06:44:26 So you can't predict the next free byte, so to speak. 06:44:36 Not byte, but allocation in general. 06:44:53 Well, this is written with the eventual idea of running on bare metal, in which case that will all be under my control. 06:44:59 Ah. Right. 06:45:03 But I'm not actually using OS calls to allocate and deallocate memory. 06:45:14 I just give the thing a bunch of memory when it loads, and it does with it what it wants. 06:45:21 I never presume that two heap pages will be adjacent. 06:46:36 Among other things, this means that something like CREATE FOO 256 ALLOT is dangerous. 06:46:51 There might not be 256 bytes left in the current definition page. 06:47:28 I expect I'll have another, more complete, dynamic memory system eventually. 06:47:52 But I didn't want the complexity of a full-on heap with variable block size, garbage collection, and so on in my core system. 06:48:22 What I imagine I'll ultimately end up with is this simple heap growing from the bottom of free RAM, and a "real" heap growing down from the top. 06:49:32 In which case the above code would be done as 256 FANCY_HEAP CONSTANT FOO 06:50:07 Maybe with some sugar, so it was 256 BUFFER FOO 06:50:35 Then later I could say FOO FREE or something. 06:50:50 Or -BUFFER 06:51:17 When I first wrote this thing I was thinking in terms of essentially unlimited MacOS memory. 06:51:38 I used 64k heap page size, just grabbed one whenever I wanted one, and also had each process residing fully within one page. 06:51:59 But that model doesn't port to small memory environments well, like this little Cortex M4 system I've got (192k RAM). 06:52:26 So I reworked it such that processes use several pages for various purposes, and such that header and definition spaces can grow to new pages. 06:53:05 Task blocks, which contain things like the stacks, the task variables SP0 and RP0, and so on, still exist in single heap pages. 06:53:15 So that's what sets the lower bound on my heap page size. 06:54:11 Right now it's set to 1k. So the "raw" blocks are 1kB + two cells (for the linked list); when I call HEAP I get back an address and I know there are 1024 bytes there for me to use. 06:55:09 Error recovery calls (HEAP) to get snapshot pages, and (-HEAP) when it doesn't need them any more. 06:55:25 I don't want those pages put on the list, because the error recovery is following that list to know what to snapshot. 06:55:37 It would never finish if those went on the list - it would wind up snapshotting its snapshot. 06:55:52 It maintains its own list of things it has to undo later. 07:00:54 Ah, so you snapshot everything so that you can restore the system exactly as it was before the failure. 07:02:34 That's right. 07:02:41 So I can restore the *process*. 07:02:45 I don't snapshot the whole system. 07:02:54 Right now it's essentially the same thing, because I have only one process. 07:03:12 But the heap management variables lie out side the process, and therein is the issue. 07:03:39 But it's quite a sledge hammer - it keeps the process sane almost completely. 07:04:00 And I get the benefit of not having an error toally reset my data stack. 07:04:04 totally 07:04:17 It resets the return stack - it restores the process pages and then runs QUIT. 07:04:32 But any items that were on the data stack at the beginning of the line are still there after the error recovery. 07:05:14 I wanted to be able, for example, to do a bunch of interactive things - say I just spent 20 lines setting up the stack for some purpose. 07:05:22 I didn't want to have a casual typo nuke that work. 07:05:57 So generally speaking I can just use my command history to cursor back to the bad line, fix it, and hit enter. 07:06:24 Another thing that will be an issue will be communicating processes. 07:06:29 Right. Many implementations nuke the data stack when an error happens. 07:06:32 Kind of annoying. 07:06:38 If A sends B a message, then A encounters an error, there's no good way to unroll the message send. 07:06:49 Not only could be have already received it, it could have already acted on it. 07:06:58 Ah, so you're adding processes as well? Do you keep track of them in memory? 07:07:00 So it's not "perfect." But it's really quite good. 07:07:15 Does your target have multiple cores? 07:07:17 Yes, there will be a process ring. 07:07:28 Doubly linked, with a system variable providing an access point to it. 07:07:41 Well, I don't have a certain target. 07:07:44 My Mac has four cores. 07:07:51 This Cortex M4 board has only one. 07:08:00 I have a Linux system at the office that has 32 cores. 07:08:24 But even on a one core system I'd be able to write applications structured as communicating processes. 07:08:37 https://en.wikipedia.org/wiki/Flow-based_programming 07:11:40 Or I might just decide whether to use multiple processes based on my target. 07:11:45 I want the ability to do it. 07:12:04 One of my more common uses of that might be to just have multiple sessions of interactive work going. 07:12:13