This is an implementation of #39 and limited to x86/x64 for Windows and Linux ABIs for time being.
There are some working toy examples in test/test.lua and test/intrinsic_spec.lua of the current API. JIT support for support for Vector register will be left as NYI because it needs various change to the JITs systems. If you feeling brave you can try out a experimental branch with JIT vector support.

An intrinsic can either be single machine instruction that LuaJIT might have some specialized understanding of or an opaque blob of 1 or more machine instructions that may be user supplied. Intrinsics will behave like a callable function in the interpreter There argument order will be the same order that input registers were declared in the register list.

API

ffi.cdef([[int32_t popcnt(int32_t n) __mcode("f30fb8rM");]])

Declaring an vector opcode intrinsic with immediate control byte

ffi.cdef([[
  typedef float float4 __attribute__((__vector_size__(16)));
  float4 shufps_rev(float4 v1, float4 v2) __mcode("0FC6rMU", 0x1b);
]])

Declaring an opcode with both a prefix and immediate byte, that takes an address and has memory side effects.

ffi.cdef[[void atomicadd1(int32_t* nptr) __mcode("830mIUPS", 0xF0, 0x01);]]

Running intrinsics in the interpreter

To allow calling intrinsics in the interpreter an internal wrapper function is generated using part of the existing JIT engine, in theory the full JIT engine could be used by generating IR instead of using the raw emit system but would probably require lots fixes where its assumed the code is being generated for a trace. The wrapper is called with two pointers the first is the input context structure that contains the values(or pointers for vectors) to the values of the input registers and the second is the Lua stack to write the results to . After the intrinsics code in the wrapper has run the wrapper writes output registers directly to the Lua stack if they are 32bit signed numbers otherwise it copies the output registers into the pre-created(before the wrapper is called) cdata that's on the Lua stack.

Intrinsics in the JIT

Three new IR instructions are added for intrinsics:

• IR_INTRN Representing the intrinsic. It is considered to have side effects with respect to DCE and as a potential load for DSE. If the intrinsic is flagged as having memory side effects a memory barrier(XBAR) will be emitted by the intrinsic trace recorder after the intrinsic. op1 points to an ordered chain CARG opcodes holding the values of the input registers and op2 points to the intrinsic ctype id.
• IR_ASMRET that is used to represent any extra output registers of an intrinsic and apply any register shuffling that will be needed for fixed registers. op1 points to the ASM op that the output values belongs to. ASMRET is skipped for the first dynamic output register of opcode intrinsics.
  op2(literal) holds the fixed register id that the output value gets written to.
  ASMRET for fixed registers have matching register hints set in register hint prepass.
• IR_ASMEND Is used as the tail of the linked list of ASMRET instructions and the starting IR instruction to emit a intrinsic with multiple output registers.

Design notes

The mcode api/system was generalized to allow more than one mcode area since the existing JIT one is flushed when a full trace flush happens, while the generated wrappers need to stay around until state is closed. In theory the FFI callback stubs could also live in this mcode area as well instead of living in fixed size memory.
Currently arguments passed to an intrinsic in the interpreter are handled using a data drive approach in which they are converted and packed into a context like the FFI system uses to call C functions. If the input values were treated as strongly typed(direct ctype id match for cdata or built-in Lua type) the need to save and load input values into the context could be skipped by the wrapper directly loading the values off the Lua stack and moving them into registers.
Currently the only way to express memory side effects that a intrinsic does is XBAR when all might be needed is a fake store of particular size that the pointer aliasing system understands also see previous discussion of how s/l/mfence could work.

Tasks

• Signed 32 bit input and out register.
• double/float input and out registers.
• Box non 32 integer/fp output registers into cdata.
• 128/256 bit vector registers(interpreter only).
• Handle mcode area reallocations when generating wrapper.
• Reuse parts of the FFI c parsing system to fully declare a intrinsic in string form(__mcode(opstring, prefix, immediate)).
• Dynamic ModRM generation for input/output registers of single opcode intrinsics that are flagged to support it.
• Option to treat large user intrinsics as callable function in the jit with custom register calling convention and know modified registers.
• support for immediate control byte when constructing opcode intrinsics.