ns3 question about global variable in RIOT

Hi all,

Daniel Camara from the ns3 team (in cc) asked me a question that I thought I’d better relay here to gather the most informed opinions:

“Is there any conceivable way for all global variables in RIOT to be declared as thread local? If this could be done adaptation with ns-3 would be MUCH easier.”

Any opinion on this question? Or on how to better formulate the question?

Caveat: Daniel does not know RIOT code well, but knows a lot about ns3, and about porting to ns3 (as he was heavily involved in the porting of Contiki to ns3).

Best,

Emmanuel

Hi Daniel & all,

Daniel Camara from the ns3 team (in cc) asked me a question that I thought I'd better relay here to gather the most informed opinions:

"Is there any conceivable way for all global variables in RIOT to be declared as thread local? If this could be done adaptation with ns-3 would be MUCH easier."

Any opinion on this question? Or on how to better formulate the question?

It would be possible to extend all of RIOTs global variables with a macro like RIOTGLOBAL and have that expand to _thread for a ns3 port. Not really nice, but we could do that.

What I don't really see is the performance gain. Maybe you can shed some light here. Thread-local storage requires the threading library to update all global variables and the context on every thread switch. Not using threads on the other hand would require the system scheduler to update the context. To me it appears that there is overhead involved in both possibilities that probably cancel each other out.

Using processes instead of threads has the benefit of separated address spaces though. That means that memory errors in RIOT would not lead to the whole simulation crashing.

Furthermore I suspect running RIOT as a thread might lead to conflicts with RIOTs own pthread implementation.. maybe someone from that domain can say something about this.

Cheers, Ludwig

Hi Judwig,

To be honest with you, I am not even worried about the performance problems linked to the thread/process trade-off. Even though we, yes, that may play a significant role.

My point is more in terms of “easy to implement”, ideally, if RIOT wasn’t dependent on any globals we could model the riot nodes as objects in a C++ framework, and all the iteration among nodes could be as simple as function calls. However, to minimize changes, the second easiest way would be to have all the nodes modeled as threads. The advantage is that we can get rid of the biggest hassle we have with process that is manipulating the shared memory area. Other advantage is that the debugging process is somehow simpler.

It is not MANDATORY, I was just curious. (OK, yes, you can say I am lazy, and that is true :)… but It is more polite to say… I would like to accomplish my task, with the minimum effort possible. I guess we can say it is an “optimization” problem to where we want to minimize effort. :slight_smile:

However, what you said kind of scared me a little, what is the MTBF of Riot? If crashing is more or less expected… then your concern is more than justifiable and process is maybe the way to go. Even though in a simulation, if a machine crashes, and it wasn’t expected… that may pollute the results and the simulation should be aborted anyway… well at least a BIG warning should be raised… unless it is the homework of networks 101, in that case… no one cares.

Best regards…

Daniel

Hi Daniel,

To be honest with you, I am not even worried about the performance problems linked to the thread/process trade-off. Even though we, yes, that may play a significant role.

My point is more in terms of "easy to implement", ideally, if RIOT wasn't dependent on any globals we could model the riot nodes as objects in a C++ framework, and all the iteration among nodes could be as simple as function calls.

I have trouble imagining how that would work - even without global variables. RIOT has a preemptive scheduler and busy loops. That is: RIOT threads can depend on preemption.

However, to minimize changes, the second easiest way would be to have all the nodes modeled as threads. The advantage is that we can get rid of the biggest hassle we have with process that is manipulating the shared memory area. Other advantage is that the debugging process is somehow simpler.

It is not MANDATORY, I was just curious. (OK, yes, you can say I am lazy, and that is true :)... but It is more polite to say.... I would like to accomplish my task, with the minimum effort possible. I guess we can say it is an "optimization" problem to where we want to minimize effort. :slight_smile:

OK then - like I said - it is possible. I think the bigger question is about the feasibility regarding threading implementation conflicts. (Anyone willing to contribute some insight on the possibility of running RIOT inside a pthread here?)

However, what you said kind of scared me a little, what is the MTBF of Riot?

That depends on how fresh the code is. I mean - the network stack for example is under heavy development at the moment and during a phase like that unchecked NULL pointers seem to slip through every once in a while. Of course we will get to a point where the stack is in sane state, but I assume one might want to use the simulator to verify some implementation aspects during development.

If crashing is more or less expected... then your concern is more than justifiable and process is maybe the way to go. Even though in a simulation, if a machine crashes, and it wasn't expected.... that may pollute the results and the simulation should be aborted anyway.... well at least a BIG warning should be raised.... unless it is the homework of networks 101, in that case.... no one cares.

Of course, the results will be spoilt. But I don't know ns3 - if it's no big deal if it crashes that's fine with me as well. If it can cause damage (e.g. by corrupting files one might still want to have used afterwards) that might be an argument for separate processes.

Cheers, Ludwig

Hi Ludwig,

I have trouble imagining how that would work - even without global variables. RIOT has a preemptive scheduler and busy loops. That is: RIOT threads can depend on preemption.

  Humm, scheduler it is easy, in fact, we will NEED to threat any way by reflecting the Riot scheduler over the ns-3 scheduler, the busy waits... well... that is another history... it would surely be a problem :(... don't know I would need to think more over it.

OK then - like I said - it is possible. I think the bigger question is about the feasibility regarding threading implementation conflicts. (Anyone willing to contribute some insight on the possibility of running RIOT inside a pthread here?)

> However, what you said kind of scared me a little, what is the MTBF of > Riot?

That depends on how fresh the code is. I mean - the network stack for example is under heavy development at the moment and during a phase like that unchecked NULL pointers seem to slip through every once in a while. Of course we will get to a point where the stack is in sane state, but I assume one might want to use the simulator to verify some implementation aspects during development.

That is normal, and it is the price to pay to use the latest version of an ongoing project. If this is the case, this does not scares me by no means :).

Of course, the results will be spoilt. But I don't know ns3 - if it's no big deal if it crashes that's fine with me as well. If it can cause damage (e.g. by corrupting files one might still want to have used afterwards) that might be an argument for separate processes.

  Ok Ludwig, there are two mind states that need to be clear and distinct

one from the other. There is "yours", as RIOT developer that want to test your development and verify the behavior of RIOT over a more complex network setup. That is valid and, I guess, it is the why you, as community, would be interested in the adaptation. The second state of mind is the guy who wants to use this framework as a tool to test his own algorithms and theories. Some one that will use RIOT+ns-3 as as a medium to see other things, I am not interested in the tool itself it should be transparent to me. I mean, if the tool does not work correctly it is like you said to me... "Look I have this hammer here and you can use..... but it has some wholes on it!!! So it will not work every time!". I may use your hammer, if I have NO other tool... but just until I find a rock that is nearby :).

    Best regards...

                     Daniel

Hi Daniel,

[…]

Of course, the results will be spoilt. But I don’t know ns3 - if it’s no big deal if it crashes that’s fine with me as well. If it can cause damage (e.g. by corrupting files one might still want to have used afterwards) that might be an argument for separate processes.

Ok Ludwig, there are two mind states that need to be clear and distinct one from the other. There is “yours”, as RIOT developer that want to test your development and verify the behavior of RIOT over a more complex network setup. That is valid and, I guess, it is the why you, as community, would be interested in the adaptation. The second state of mind is the guy who wants to use this framework as a tool to test his own algorithms and theories. Some one that will use RIOT+ns-3 as as a medium to see other things, I am not interested in the tool itself it should be transparent to me. I mean, if the tool does not work correctly it is like you said to me… “Look I have this hammer here and you can use… but it has some wholes on it!!! So it will not work every time!”. I may use your hammer, if I have NO other tool… but just until I find a rock that is nearby :).

We’re on the same wavelength here, and I’m not sure it is worth discussing too much this question.

While the core of RIOT is extremely stable, it’s just that one cannot expect a whole OS + the bleeding edge of full network stacks under development to be as stable as a 1k lines of code implementing a simple network protocol on top of ns-3.

For instance anybody testing Contiki+ some new networked application on ns-3 has realistically exactly the same concerns, which is inherent to this type of approach. If it made sense to port Contiki to ns-3 then it makes sense to port RIOT to ns-3. That’s a simpler way to see it :wink:

So let’s just do it and focus on the “how”. Other questions are irrelevant in this context, in my opinion.

Does it make sense?

Cheers,

Emmanuel

Hi Emanuel,

Don’t get me wrong, I am not judging, complaining, or something like that, just stating the fact that we have two VERY distinct clients here, just that! Even though this obvious for me, it needs to be clear to all. Well, I guess, even if the only clients were RIOT developers, it would already worth the effort… not that cool, we could do better, but probably would already pay the price… and the ways to implement the adaptation could be highly dependent on it.

So yes I agree with you, we need to focus on hows, but I also partially disagree because this specific point may impact the way the adaptation will be made. If you decide the public is just the RIOT development team… and you don’t care for the second class of users (what is completely fair), more “home made” and code dependent options could be considered… , for example, we wouldn’t need something that is fairly independent of versions of ns-3. You could fix your code over, lets say, version 19 and that is it would be more than enough for you.

Once more, don’t get me wrong! Not judging, just trying to understand :).

Hi,

I am confused - I was not arguing for forcing buggy code on people. I just wanted to say that it might be worth the extra mile (the extra work of supporting processes instead of threads). Anyway - let's find out if threads are possible at all first and talk about this if it's an option at all.

Regarding busy loops - I guess it might be an option to say that the ns-3 port does not support those, it's not like they are an essential part of RIOT (at least not to my knowledge, I could only find occurrences of busy loops in cpu specific code).

Cheers, Ludwig

Hi!

Regarding busy loops - I guess it might be an option to say that the ns-3 port does not support those, it's not like they are an essential part of RIOT (at least not to my knowledge, I could only find occurrences of busy loops in cpu specific code).

And these could/should be probably replaced by some interrupt driven implementations. The only busy loop that can be definitely not be replaced by something interrupt driven that I remember by heart is the delay function to let the LEDs blink at startup (because hwtimer is not initialized then) - and I guess that's a non-blocker. :wink:

Cheers, Oleg

Hi Daniel

Even though I have a close relation with the ns-3 community, and I have no information whatsoever, on changes that could affect the adaptation, I can’t guarantee this will not happen in the future. The only way to be fully independent of changes would be to fix a version and go with that for ever, and NO, I don’t think it is a good idea.

The other way to be independent is to create a thin specific control layer, with a generic interface. If something change in any one of the sides (RIOT is also a living project, things will change for sure in the future) just the thin implementation need to be changed, the rest will stay the same.

As I understand the adaptation work should be done is as it was a Middleware between RIOT and ns-3. For RIOT ns-3 will be like a new architecture, a new sensor where it will be running and for ns-3, RIOT will be just a new kind of node, one should be as blind about the behavior of the other as possible. Something like:

Hi Daniel, I think what you suggest makes sense. RIOT and ns3 are living projects, so will evolve and it is not reasonable to fix one version of any to achieve the port. Fortunately, the core of RIOT and of ns3 should be stable enough for it to be worth it to design the thin layers-based middleware you propose. Let’s refine this idea, I think this is the right direction. Cheers Emmanuel