Climbing Elbrus - Reconnaissance. Technical Part 2. Interrupts, exceptions, system timer

We continue to explore Elbrus by porting Embox to it.

This article is the second part of the technical article on the architecture of Elbrus. The first part dealt with stacks, registers, and so on. Before reading this part, we recommend to study the first one, since it tells about the basic things of the Elbrus architecture. This part deals with timers, interrupts and exceptions. This, again, is not official documentation. She should contact the developers of Elbrus in the MCST .
Starting the study of Elbrus, we wanted to quickly start the timer, because, as you understand, preemptive multitasking does not work without it. For this, it seemed enough to implement the interrupt controller and the timer itself, but we ran into unexpected expected difficulties, but without them. We began to look for debugging opportunities and found out that the developers took care of this by entering several commands that allow us to raise various exceptions. For example, you can generate a special type exception using the PSR (Processor Status Register) and UPSR (User processor status register) registers. For PSR, the exc_last_wish bit is the exc_last_wish flag for generating an exception when returning from the procedure, and for the UPSR is exc_d_interrupt, this is the pending interrupt flag generated by the VFDI operation (Check pending interrupt flag).

The code is as follows:
')

#define UPSR_DI (1 << 3) /*   .h  */ rrs %upsr, %r1 ors %r1, UPSR_DI, %r1 /* upsr |= UPSR_DI; */ rws %r1, %upsr vfdi /*      */ 

Launched. But nothing happened, the system was hanging somewhere, nothing was output to the console. Actually, this is what we saw when we tried to start an interruption from the timer, but then there were many components, but it was clear that something interrupted the sequential execution of our program, and control was transferred to the exception table (in terms of the Elbrus architecture, it’s more correct to speak not about the table interrupts on exception table). We assumed that the processor nevertheless produced an exception, but there, where he transferred control, is some kind of “garbage”. As it turned out, he transfers control to the very place where we put the Embox image, which means there was an entry point - the entry function.

To check we did the following. Brought the entry counter to entry (). Initially, all CPUs start with interrupts turned off, enter entry (), after which we leave only one core active, all others go into an infinite loop. After the counter has reached the number of CPUs, we consider that all subsequent hits on the entry are exceptions. I remind you that it used to be as described in our very first article about Elbrus

  cpuid = __e2k_atomic32_add(1, &last_cpuid); if (cpuid > 1) { /* XXX currently we support only single core */ while(1); } /* copy of trap table */ memcpy((void*)0, &_t_entry, 0x1800); kernel_start(); 

Did so

  /* Since we enable exceptions only when all CPUs except the main one * reached the idle state (cpu_idle), we can rely that order and can * guarantee exceptions happen strictly after all CPUS entries. */ if (entries_count >= CPU_COUNT) { /* Entering here because of expection or interrupt */ e2k_trap_handler(regs); ... } /* It wasn't exception, so we decide this usual program execution, * that is, Embox started on CPU0 or CPU1 */ e2k_wait_all(); entries_count = __e2k_atomic32_add(1, &entries_count); if (entries_count > 1) { /* XXX currently we support only single core */ cpu_idle(); } e2k_kernel_start(); } 

And finally, we saw the reaction to the input to the interrupt (just with the help of printf, the line was output).

Here it is worth explaining that initially in the first version we hoped to copy the table of exceptions, but first, it turned out that it is located at our address, and second, we did not manage to make a correct copy. I had to rewrite the linker scripts, the entry point into the system, and of the interrupt handler, that is, the assembler part was needed, about it a bit later.

This is how the part of the modified linker script now looks like:

 .text : { _start = .; _t_entry = .; /* Interrupt handler */ *(.ttable_entry0) . = _t_entry + 0x800; /* Syscall handler */ *(.ttable_entry1) . = _t_entry + 0x1000; /* longjmp handler */ *(.ttable_entry2) . = _t_entry + 0x1800; _t_entry_end = .; *(.e2k_entry) *(.cpu_idle) /* text */ } 

that is, we removed the entry section for the exception table. There is also a cpu_idle section for those CPUs that are not used.

This is what the login function for our active kernel looks like on which Embox will run:

 static void e2k_kernel_start(void) { extern void kernel_start(void); int psr; /*    CPU “” */ while (idled_cpus_count < CPU_COUNT - 1) ; ... /*     ,     */ e2k_upsr_write(e2k_upsr_read() & ~UPSR_FE); kernel_start(); /*   Embox */ } 

Well, according to the VFDI instructions, an exception has been made. Now you need to get his number to make sure that this is the correct exception. For this purpose there are TIR (Trap Info registers) interrupt information registers in Elbrus. They contain information about the last several commands, that is, the final part of the trace (trace). Trace is collected during program execution and “frozen” when entering an interrupt. TIR includes the younger (64 bits) and older (64 bits) parts. The low word contains exception flags, and the high word is a pointer to the instruction that led to the exception and the number of the current TIR'a. Accordingly, in our case, exc_d_interrupt is the 4th bit.

Note we still have some misunderstanding regarding the depth (count) of TIRs. The documentation provides:

“The TIR memory depth, i.e. the number of trap info registers, is determined by
TIR_NUM macro, equal to the number of processor pipeline stages required for
issuing all possible special situations. TIR_NUM = 19; "

In practice, we see depth = 1, and therefore we use only the TIR0 register.

Specialists at the MCST explained to us that everything is correct, and for “accurate” interruptions there will be only TIR0, and for other situations there may be more. But since so far we are talking only about interrupts from the timer, this does not prevent us.

Ok, now let's see what is needed for the correct entry / exit from the exception handler. In fact, it is necessary to save on input and restore on output 5 of the following registers. The three control transfer preparation registers are ctpr [1,2,3], and the two cycle control registers are the ILCR (Cycle Counter Original Value Register) and LSR (Cycle Status Register).

 .type ttable_entry0,@function ttable_entry0: setwd wsz = 0x10, nfx = 1; rrd %ctpr1, %dr1 rrd %ctpr2, %dr2 rrd %ctpr3, %dr3 rrd %ilcr, %dr4 rrd %lsr, %dr5 /* sizeof pt_regs */ getsp -(5 * 8), %dr0 std %dr1, [%dr0 + PT_CTRP1] /* regs->ctpr1 = ctpr1 */ std %dr2, [%dr0 + PT_CTRP2] /* regs->ctpr2 = ctpr2 */ std %dr3, [%dr0 + PT_CTRP3] /* regs->ctpr3 = ctpr3 */ std %dr4, [%dr0 + PT_ILCR] /* regs->ilcr = ilcr */ std %dr5, [%dr0 + PT_LSR] /* regs->lsr = lsr */ disp %ctpr1, e2k_entry ct %ctpr1 

Actually, that's all, after exiting the exception handler, you need to restore these 5 registers.

We do this with a macro:

 #define RESTORE_COMMON_REGS(regs) \ ({ \ uint64_t ctpr1 = regs->ctpr1, ctpr2 = regs->ctpr2, \ ctpr3 = regs->ctpr3, lsr = regs->lsr, \ ilcr = regs->ilcr; \ /* ctpr2 is restored first because of tight time constraints \ * on restoring ctpr2 and aaldv. */ \ E2K_SET_DSREG(ctpr1, ctpr1); \ E2K_SET_DSREG(ctpr2, ctpr2); \ E2K_SET_DSREG(ctpr3, ctpr3); \ E2K_SET_DSREG(lsr, lsr); \ E2K_SET_DSREG(ilcr, ilcr); \ }) 

It is also important not to forget, after register recovery, to call operation DONE (Return from the hardware interrupt handler). This operation is needed in particular in order to correctly handle the interrupted control transfer operations. We do this using a macro:

 #define E2K_DONE \ do { \ asm volatile ("{nop 3} {done}" ::: "ctpr3"); \ } while (0) 

We actually do the return from the interrupt directly in the C code using these two macros.

  /* Entering here because of expection or interrupt */ e2k_trap_handler(regs); RESTORE_COMMON_REGS(regs); E2K_DONE; 

External interrupts

Let's start with how to enable external interrupts. In Elbrus, APIC (more precisely, its counterpart) is used as an interrupt controller, Embox already had this driver. Therefore, it was possible to pick up a system timer to it. There are two timers, one kind of very similar to the PIT , the other LAPIC Timer , also quite standard, so it makes no sense to talk about them. Both that, and that looked simply, and that and that in Embox already was, but the driver of the LAPIC-timer looked more promising, besides, the implementation of the PIT timer seemed to us more non-standard. Consequently, it seemed easier to finish. In addition, in the official documentation, the APIC and LAPIC registers were described, which differed slightly from the originals. It makes no sense to bring them, because you can see in the original.

In addition to enabling interrupts in APIC, you must enable interrupt handling through the PSR / UPSR registers. Both registers have permission flags for external interrupts and nonmaskable interrupts. But it is very important to note here that the PSR register is local to the function (this was mentioned in the first technical part ). And this means that if you set it inside a function, then when you call all subsequent functions, it will be inherited, but when you return from the function it will return its original state. Hence the question, but how to manage interrupts?

We use the following solution. The PSR allows you to enable management via UPSR, which is already global (which is what we need). Therefore, we allow management via UPSR directly (important!) Before the Embox kernel login function:

  /* PSR is local register and makes sense only within a function, * so we set it here before kernel start. */ asm volatile ("rrs %%psr, %0" : "=r"(psr) :); psr |= (PSR_IE | PSR_NMIE | PSR_UIE); asm volatile ("rws %0, %%psr" : : "ri"(psr)); kernel_start(); 

Somehow by chance, after refactoring, I took and rendered these lines into a separate function ... And the register is local to the function. It is clear that everything is broken :)

So, in the processor, everything you need is supposedly turned on, go to the interrupt controller.

As we already sorted out above, information about the exception number is in the TIR register. Further, the 32nd bit in this register reports that an external interrupt has occurred.

After turning on the timer followed a couple of days of torment, since no interruption could be obtained. The reason was funny enough. In Elbrus 64-bit pointers, and the address of the register in the APIC is in uint32_t, so we used them. But it turned out that if you need, for example, to lead 0xF0000000 to a pointer, then you will receive not 0xF0000000, but 0xFFFFFFFFF0000000. That is, the compiler will extend your unsigned int sign.

Here, of course, it was necessary to use uintptr_t, therefore, as it turned out, in the C99 standard this kind of type conversion was implementation defined.

After we finally saw the raised 32nd bit in TIR, we began to look for how to get the interrupt number. It turned out to be quite simple, although not at all like on x86, this is one of the differences between LAPIC implementations. For Elbrus, in order to get the interruption number, you need to climb into the special LAPIC register:

  #define APIC_VECT (0xFEE00000 + 0xFF0) 

where 0xFEE00000 is the base address of the LAPIC registers.

That's all, it turned out to pick up the system timer and LAPIC timer.

Conclusion

The information given in the first two technical parts of the article about the Elbrus architecture is enough to implement hardware interrupts and preemptive multitasking in any OS. Actually, the above screenshots show this.



This is not the last technical part about the architecture of Elbrus. Now we are mastering memory management (MMU) in Elbrus, we hope we will soon tell you about it. We need this not only for the implementation of virtual address spaces, but also for normal work with peripherals, because through this mechanism you can disable or enable caching of a specific area of ​​the address space.

Everything that is written in the article can be found in the Embox repository. You can also build and run, if of course there is a hardware platform. True, this requires a compiler, and it can only be obtained in the MCST . Official documentation can be requested there.