wip,src: add utf8 consumer/validator

[chrisdickinson]

WIP. Opening this now to double check that this is headed in the right direction.

This is based on utf-8-validate. The primary differences are the use of clz (vs. a lookup table) to compute the number of extra bytes and the introduction of the error/glyph callbacks.

The Utf8Consume function will iterate over valid glyphs, calling a provided OnGlyph callback and an OnError callback as necessary. Provided error strategies are "Halt" and "Skip" – which either halts the consumer at the first error, or skips past them as appropriate.

The strategies were added to accommodate the desire to build a utf8-to-utf16 translator as part of this work.

[bnoordhuis]

Style: no C-style casts (here and elsewhere.)

[chrisdickinson]

Fixed. Sorry about that, missed while porting.

[bnoordhuis]

0xC2? Shouldn't that be 0xC0?

[chrisdickinson]

Hm. I'll look into that – the original version has it as 0xC2.

[piscisaureus]

The msvc "equivalent" is _BitScanReverse. https://github.com/libuv/libuv/blob/3346082132dc2ff809dfd5d25d451c0b33905f53/src/win/tty.c#L1442

Edit: you already knew :) sorry

[bnoordhuis]

I think what you want here can be implemented portable and reasonably efficient using the following:

inline uint32_t log2(uint8_t v) {
  const uint32_t r = (v > 15) << 2;
  v >>= r;
  const uint32_t s = (v > 3) << 1;
  v >>= s;
  v >>= 1;
  return r | s | v;
}

inline uint32_t clz(uint8_t v) {
  // clz(0) == 7.  Add a zero check if that's an issue.
  return 7 - log2(v);
}

[bnoordhuis]

Forgot to mention, the behavior of static_cast<int>(...) << 24 is implementation-defined on platforms where ints are 32 bits (i.e. all of them.) You're not allowed to shift values into the sign bit.

[bnoordhuis]

Shouldn't this mask off the high bits? Also, when is extrabytes == 5 for valid UTF-8?

[bnoordhuis]

Real Programmers(TM) write this as (glyph & 0x7800) != 0x5800 :-)

[piscisaureus]

Real Programmers(TM) write this as (glyph & 0x7800) != 0x5800 :-)

Compilers are pretty good in replacing idiomatic range comparisons with bit hacks. Have you checked whether it's really necessary to have these ... ?

[bnoordhuis]

Har, har.

(But it's true that both clang and gcc manage to pull it off.)

[jbergstroem]

Now that #1199 landed -- could we perhaps add unit tests for this?

[chrisdickinson]

Nixed clz support, since a quick benchmark showed that @bnoordhuis' implementation was faster.

[bnoordhuis]

Style issue: the first argument goes on the same line as the function name and the other arguments should line up below it. The only time that's deviated from is when the 80 column limit is exceeded.

[bnoordhuis]

I second @jbergstroem's sentiment on unit tests. :-)

[petkaantonov]

Also always interesting to run on an utf-8 decoder is the utf8 decoder stress test