WPF. Sound run-time loading Unity

A little more than a week has gone by and more programming has been done. This time I was requested to put sound into Unity. Sound in Unity is normally not a problem. It’s just a matter of saying “Hey Unity, I have this AudioClip, drag it onto the AudioSource and you’re done.

What I didn’t account for was when the user wants to load a custom clip, that you need to load it into memory first. All of a sudden it’s not as easy as with images. There are loads of codecs and some support streaming, others do not.

Loading sounds during run-time

The solution to loading sounds during run-time was to use the WWW class that exists in Unity. Basically what you do is use:
WWW www = new WWW(“file://pathAndFilename”);
AudioClip clip = http://www.GetAudioClip(false, false);  // Worth noting, last false is if you want it streamable or not.
Then the solution that worked best for me was to use Getcomponent<AudioSource>().PlayOneShot(clip);

Since this was part of the animation that I did I just expanded the class and it seemed to work. I did get some complaints, so I need to look into why that is and how I can fix it. Locally however it works flawlessly. This however only worked for .ogg files. Loading .wav I have not tested.

WPF – Windows Presentation Foundation

The second thing I wanted to discuss, and this is actually the biggest topic is the WPF. At first it was a true pain in the ass to say it bluntly. Considering however that I started today to see how it works, I’m happy with the progress. I was able to draw the application and get it to look how I wanted it to look.

Here’s a screenshot of the Stream Overlay I have recreated today:

So far I have only created the graphics layout with some help of a friend (Henrik) who gave me tips on what colors that matched each other. Of course the UI needs to be streamlined a little bit. I have some spacing here and there I need to fix, but that is a small issue.

Working with WPF

As I said in the beginning, working with WPF was a not particularly fun. I did get that most of it is about working with XML in the editor. What I have learned however is to use styles that you can modify the way you want. Combo boxes for instance was not easy to get to look like the rest of them. But luckily there’s a way to get a whole bunch of XML generated for you. Once you figure out how the different functionalities work it’s not too hard. Besides the editor in Visual Studio 2015 works well. There were a couple of instances where I was sure that I had messed up when editing the XML. But luckily you can see what line is causing the problem and try to figure out the problem from there.

In my opinion it’s not as easy as working with the old Windows Forms, but it looks SO much better. It looks sleek and more inviting. I will try to continue develop both in tandem, but my main focus will most likely be to use the new WPF. Also I have to say as usual, Google is vital in finding errors and problems that might occur. Even if the error or problem isn’t the same you get a hint how to make it work.

One example of this was when I started out, I managed to delete the <Grid></Grid> tags and I couldn’t get any of the Windows controls to work. But after checking the error, I found a place that described that problem in a different context and I was able to see that <Grid> tags were missing for me.

From what I gather as well, the way WPF works is the same as Android works, it’s slightly different, but it’s very similar. So that bodes well when I want to look at developing an android interface for this at some point. I have looked at it before, just never got around to really look at the Android GUI.

Last but not least

Lastly though it is worth noting that what I have is just the GUI. It’s not working. The only two things that actually work at this point is to drag the window around and closing it by clicking the cross on the top right corner. So I expect that it isn’t as easy to just copy code over from the other project, but time will tell.

Next time I will hopefully still have my hair intact and that the code I have created for the Windows Forms project is more or less portable as is. If not see you hairless and frustrated next time.

Overlay – setbacks and progress

I didn’t know that it has been this long since I last blogged anything, but I’ve been busy with the program I’m working on. There’s been some progress, but also some setbacks. Mostly the setbacks have been caused by unexpected behaviors and poor planning. Yes, poor planning, but in all honesty most of this program started with just a test and then evolved into a test. Then I showed that test to someone and I got inspired to continue working on it. Which means that before I started I didn’t really plan on the amount of features that the program currently has. The program parts currently looks like the images below.

Overlay

Control Panel (Yes, I am aware of the misalignment between the buttons)

Setbacks

Since I haven’t had much time to plan anything, some of the programming I did in the early stages was done just to get things to work. One example of  this was before I discovered serializing. Serializing in C# is very easy, in fact it’s so easy to do that when creating files in the future, that’s the method I prefer to do. There’s in principle 3 steps, here’s one snippet from code I am currently using:

XmlSerializer formatter = new XmlSerializer(typeof(List<Preset>));
Stream stream = new FileStream(settingFilename, FileMode.Create, FileAccess.Write, FileShare.None);
formatter.Serialize(stream, listOfPresets);

But before I discovered that simple way of writing to file and reading from file, I used a far more complex way, this is an example of that using an XML reader and reading each tag:

string xmlContent = File.ReadAllText(“.\\settings.xml”);
XElement xmlElements = XElement.Parse(xmlContent);
List<XElement> xmlElementList = xmlElements.Elements().ToList();
for (int i = 0; i < xmlElementList.Count; i++)
{
List<XElement> subElements = xmlElementList[i].Elements().ToList();
if (xmlElementList[i].Name == “WindowBGColor”)
(……) And additional 50 lines of code

Quite a difference in code and how easy it is to maintain. If the class changes all I need is the [Serializable] on the classes I add and the code doesn’t have to change at all. The setback was that I needed to rewrite the code, although in the end it was less code than I started with. But that means it is easier to maintain.

The other setback I had was that when I first started I had everything in so called packages. The thought behind it was that in the beginning i wanted to create packages sent through a named pipe. The named pipe required to know the length of the data and a buffer. I created a system where I send a type of package, length and data. But I have since changed from named pipe to UDP, but the packages still worked as I wanted them to work. So I kept the name. But I was locked into packages and thus created all kinds of package this, package that, because I wanted to have the same type on receiving and sending side. This package thought crept into other parts of the program and before I knew it I had Tray, TrayPackage and TrayData. Preset, PresetData, PresetPackage. And so on and so forth.

Some of the reasoning, if you can call it that, was  that I needed to have the same datatype on the Control Panel as in the Overlay part. I have since moved away, I needed to rewrite a whole lot of code. I unified all classes having more than 1 of a similar type into one of each. So instead of Tray, Traydata and TrayPackage, I now have Tray. Instead of Preset, PresetData, PresetPackage I only have Preset. There are a few other examples, not as bad, but some had two of the same type. All of that costs time to redo and make sure that what worked before still works after changing it.

But it doesn’t stop there. Another setback was that I didn’t know I could use serialization to send data more reliably as well. Honestly I didn’t know of serialization when I started sending the “big” classes through UDP either. So I constructed a key system. The sender would construct the class into one single string. The keys were used to know where the string started and where it stopped. That way I could extract the part I needed.

The problem was that once I needed to change the class, I needed to  construct a new key. I had however created a system making it easy to extract the key, so adding keys was not really a problem, but each key made the code less readable and I needed to add it on  both sending and receiving end without any spelling errors.

Since both end didn’t have the same class, I created a way to serialized the individual parts that were the same and create a byte buffer from that and send that through to unity and decode that with a deserialize on the other end. That worked much better. The code, not so much smaller, a tiny bit, but not as much as I had hoped. Since the serialization method that is built in only works for basic types I had to break the class up in those parts, but that said, the serialization/deserialization method works well on both sides.

Progress

Despite all the setbacks and rewrite of code, which I nearly expected to do, I have made great strides. For instance now scenes are saved upon exit, but not just the images you add, but the trays as well. Rotation, scale, position are all saved. If you have a tray you wanted automated, it still is automated. Presets also clear the scene and add all the objects. If the preset has been used once before it checks the position, scale and rotation of the objects and sets them to the saved position. These positions are saved automatically.

I have also added two default presets that are a bit special. The reset scene preset that resets the scene to a default color and a default scene, which is more or less a playground to test the different functions out. It is possible to use it as a preset and it is the preset that is loaded by default.

The second part is an animator. This took far more work than I intended, at first I figured it would just be a rotating image that scaled in, then stayed on screen and then faded out while rotating out. By the time I figured I would need fade in, fade out, rotate in and rotate out timers, I figured, why not let the users set all the settings themselves? Which is what I did. Users can now decide if they want to rotate an image in, how big the image should be at the end, how fast the animation should go. At what point in time the animation should start to fade in, fade out, rotate in, rotate out, scale up and scale down. That does mean the users can mess it up, but if it doesn’t work, it shouldn’t be a problem. The animation system also works independently from presets.

Summary

Despite all the setbacks, I feel like the progression has gone forward. Now I need to take a step back. Create a plan for what I want to do forward. There are 2 features I want to implement, both I have experimented with and have managed to get working. The first feature is the miniature ticker. The miniature ticker will be used to show for instance who has donated last, latest followers and latest subscriber, to mention just a few examples. Right now the problem is that I need to figure out how to measure the length of a textmesh and there’s not too much documentation on how to do just that. Once I figure out that I can get the text to wrap around so to speak.

The second thing I want to accomplish is to do one of two. Either that the control panel sets the twitch information to Unity and that checks in certain intervals if the most recent subscriber / follower and handles that automatically. That sounds like the best solution on paper, however if twitch goes down Unity might start lagging and using a lot of cpu to try to connect. Second option is to use the control panel for that, but that means the control panel needs to be up at all times. Third option, which is probably what I am going for is to create a separate thread that sleeps for half a second and then checks. Then once there’s something new it updates the information. If that thread suddenly takes a while longer the program might take up more cpu, but at least it won’t crash or occupy the rest of the program.

If I do it on the control panel side, it wouldn’t be that problematic if something went wrong, but that also means the user need the control panel up at all times. Which is something I didn’t intend on. If the user wanted to they should be able to close the control panel. But in order to get the twitch connection to work, if i go with the control panel route, it has to stay open. First and foremost however I need to get more people to try it. Get their feedback and see what they need.

Twitchoverlay v0.3a

After my previous post, I decided to continue with the Twitch Overlay. I recently got done with a version I call 0.3a. The feature list isn’t that big, but I was able to reach a streamer and get some feedback on the program. That said if anyone reading this are streaming and want to test it out, let me know. I would love to hear from you.

One of the features requested was a tray that comes in and out of the screen. The control panel itself, the user was satisfied with, but after some testing, I found that I wanted to redesign the Control panel as well. More to get all the controls into one area, so that it is easier to keep the panel open.

Twich Controlpanel Evolution

Last week the control panel looked like this:

It also contains a lot of clutter and information. But basically I have kept that layout throughout the development process. There is too much text, but it’s difficult to tell the user what the different things are without some information. At a later point, I will refine this and give it shorter names. This is what I consider more fine tuning however, right now functionality far outweighs the text and what not.

This was the next version. Basically the same, but I played around with the webcam. Actually got that to work and got a basic chroma key to work. It’s not flawless, but for Red, Green and Blue it works fairly well. That said, I will remove that feature as OBS has a better version. Only feature my webcam code has is that you can rotate it to the degrees you want to. In both 0.1a and the first “draft” the tray functionality was missing.

The next version was 0.2a: 

Here I have started to clutter down the GUI again. This was not from feedback, but I wanted to add some functionalities to the requested feature. The trays can  have text on it. The check box is if the user wanted to use a countdown timer and the dropdown box was to see how many minutes the user wanted to have the countdown clock to show. Then the add button was to send the information to the Unity window. The turn on/off button was simply to animate the tray on and off manually.

I needed to fix some of the labels that were a bit weird, so I added some better and more text to the panel. Making version 0.21a look like this:

This is where I started to look at the user interface and see that I needed to fix this somehow. Firstly I didn’t want the interface to be this big. I only wanted to have controls so the user did not have to add stuff from that panel. So I decided to go back to the drawing board.

I removed all of the presets, I shrunk the tray settings area down. Instead I created a separate form that contains all the information I wanted to have there. Making the GUI look more like it looked like before:

Now when the user click the “Add Tray Graphics” button, they get a box up where they can enter all the settings and then the controls for the tray comes up instead. Since the tray options have more space I could add more options, so the user can now for the most part only select where they want the image to stop and the rest is automatically set up for the user. They can still use keyboard controls within the Unity window to change direction and location of text and counters. Zoom in and out graphic and text if they so desire.

The interface is still not perfect. As mentioned above, I want to streamline the interface a bit more. Since I have ticker settings as a header, I think I should remove so that it says Text Color, Background Color, Turn On/Off Text and Turn On/Off background.

But this is minor stuff at the moment, I am currently more focused on getting the features in and later I can modify and make the interface better based on feedback.

Next week I want to get the presets to work, so that the user can send the preset to Unity. Then save the preset information in Unity, so next time the preset  is sent it checks for two things. First if the preset exists the preset is loaded from Unity preset list. Image locations are restored and all settings are set the way the user wanted it to be.  Unless the user selects to delete existing preset if it already exists.