It just hasn't grown to keep up with the utterly outrageous population growth and the growth is real crude oil demand.
http://www.economist.com/countries/Nigeria/profile.cfm?folder=Profile-Economic%20StructureAgriculture is still the highest contributor to our GDP so I do not know where people get this idea that we abandoned agriculture for oil. I haven't seen much economic research on this but I think, agriculture didn't decrease as much as we would have thought. In my opinion, oil grew ridiculously faster than the growth in agriculture. The growth in agriculture and oil products did not keep up with the population growh and add the inefficiencies we see in Government, you see the poor state of the Nigerian economy.
Population growth is more of a problem than focusing on oil because after all, it's demand has been highest in the past 30 odd years. If you have it, you sell it now. Agriculture has been OK atleast in terms of domestic production, distribution, storage, processing has been bad with the failures in NEPA and the road system. This might explain why agricultural exports have fallen over the years. How will you produce cocoa when you can't transport it out of the producing region and neither can you store it without depending on generators?
By the way, we need more educated people in agriculture, farming isn't a poor uneducated man's thing.